Application Trace Files

In case Oracle Health Insurance application runs into an unanticipated error, an application trace file generates.
Application trace files contain technical information that can assist in the root cause analysis of a problem.
Typically, share application trace files when present with Oracle Health Insurance support with the product development team for analysis.

The trace files are not available for cloud customers. Instead, add trace ID to the service request for analysis.

Controlling the Target Location for Trace Files

Filesystem Location

The application writes the trace files to the filesystem by default.
The ohi.incident.rootdir property governs the actual directory.
In case this property is not set, the application writes the trace files to a /trace directory.
In that case, this is a directory that is relative to the location from where the application starts.
In case the ohi.incident.rootdir is set, the application uses the value of this property as the directory of trace files.
Note that the application creates this directory automatically, in case the directory does not exist.
The value for this property must have a directory that the O/S user that starts the application can write(and create) to.

Datafile Set Location

It is possible to have the system use datafile sets as the location of trace files.
This is especially useful in cloud-based deployments, where there is access restriction to the physical filesystem.
To have datafile set as the target location, set the following property: ohi.incident.target=datafileset.
This option writes the trace files to datafile sets having the following name pattern: _OHIIncidents_<yyyyMMdd>.
Effectively, this means that trace files end up in a specific datafile set on a daily basis.
This way trace files are accessible through the Data File Set Integration Point. It is possible to download individual trace files, or all the trace files (zipped) of a specific day (that is, a specific trace file datafile set).

Datafile Set Trace Files Retention

Activate the option to use a datafile set as the target location of trace files, these datafile sets adhere to a retention regime.
Purging the datafile set is done with the help of the datafile set purge PL/SQL procedure.
This purge process is also integrated into the Auto Purge process. If the datafile set purge process is enabled, all datafile sets will be purged automatically whenever the retention period expires.
The retention period can be set using the following API:

http://[hostName]:[portNumber]/[api-context-root]/generic/autopurgemetadata/{id}.

This property defines the number of days to retain data file sets for an application’s incidents. When the 'ohi.incident.target` property is set to datafileset, the automated schedules purges data file sets (of application’s incidents) older than the retention period. The automated scheduler is scheduled to run every midnight.

By default, an application keeps the data file sets of an application’s trace files for up to 10 days. It is possible to change the number of days through the ohi.incident.datafileset.retentionperiod property.

Identifying Application Trace Files

Every application trace file has a reference to an incident ID.
This incident ID is a system-generated piece of information, uniquely identifying a specific incident.
Incident IDs follow this pattern: [APP]-[C]-#[NR]-YYYYMMDD_HH24MMSS.

APP: the short name for the Oracle Health Insurance Application
C: the operating context; U=User Interface, W=Web/REST service, P=Processing
NR: a sequence number
YYYYMMDD : the date of the incident
HH24MMSS : the time of the incident

The end-user-facing application elements, like the UI, typically use this incident ID.
This provides a visible cue to the end-user that something went wrong in the application, and that the application writes an incident.
As such, use it further with an application operator to find out what is going on.
It is quite possible that from the trace, it is not clear what the actual cause of the problem is.
In those specific cases, typically Oracle Health Insurance application logs a support ticket, including the trace files.

The main Oracle Health Insurance application logfile will also hold an ERROR log entry when the application generates a trace file.
Following is an example of what this looks like when using the standard filesystem location as the target:

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to /home/logs/ohi/datafile/dfl-23.0.x/ohi-datafile-integration-tests/target/trace/trace-DAT-W-#1-20201020_125100.log

Setting the property ohi.incident.target to datafileset looks something like this:

The following example document is formatted for documentation.

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to https://foo.bar.invalid:9807/api/datafilesets/_OHIIncidents_20201020/datafiles/trace-DAT-W-%231-20201020_140358.log/data

Depending on the Oracle Health Insurance application, reuse the information in the ERROR log entry for other purposes.
As a canonical example: reuse this information in a database log record attaching it to a specific work item.

Example Trace File

Following is an example of a trace file.

Some formatting is applied in the following example for documentation.

------------------------------------------------------------------------------------------------
 Application : DAT (1)
 Version     : 23.0.0-SNAPSHOT (2)
 Context     : WebService (3)
 Filename    : trace-DAT-W-#1-20201020_140951.log (4)
 Date        : October 20, 2020 (5)
 Time        : 2:09:51 pm (6)
-------------------------------------------------------------------------------------------------

 Unexpected error context: No additional context (7)

 java.lang.RuntimeException: Thrown for illustration purposes only (8)
	at ...IncidentDFWriterTest.generateTraceFile(IncidentDFWriterTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:532)
...
-------------------------------------------------------------------------------------------------

 ** THREAD DUMP: (9)

"InterruptTimer" Id=41 WAITING on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)

"oracle.jdbc....BlockReleaser" Id=40 TIMED_WAITING on BlockSource$ThreadedCaching@8fa4647
	at java.lang.Object.wait(Native Method)
	-  waiting on oracle.BlockSource$ThreadedCachingBlockSource$BlockReleaser@8fa4647
	at oracle.jdbc.driver.BlockSource$ThreadedCaching.run(BlockSource.java:329)

"Timer-0" Id=39 WAITING on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)
...
-------------------------------------------------------------------------------------------------
[END] (10)

1	Shortname of the application.
2	Full version of the application.
3	Operating context at time of the incident.
4	The trace filename.
5	The date of the incident.
6	The time of the incident.
7	Optional, additional context of the incident.
8	The stacktrace of the component from the incident originated.
9	A thread dump of all the other threads that were operating at the time of the incident.
10	An indication that this is the last line of the trace file.