Application Trace Files

In case an Oracle Health Insurance system component runs into an unanticipated error an application trace file gets generated. Application trace files contain technical information that can assist in root cause analysis of a problem. Typically, application trace files when present are shared with OHI support and analyzed by product development.

Controlling the target location for trace files

Filesystem location

Application trace files by default are written to the filesystem. The actual directory for this is governed by the ohi.incident.rootdir property. In case this property is not set, trace files are written to a /trace directory. In that case, this is a directory that is relative to the location from where the application was started. In case the ohi.incident.rootdir is set, the value is used as the directory of trace files. Note that this directory will be automatically created in case it does not exist. The value for this property should hold a directory that can be (created and) written to by the O/S user that started the application.

Datafile set location

It is possible to have the system use datafile sets as the location of trace files. This is especially useful in Cloud based deployments, where access to the physical filesystem is prohibited. In order to have datafile set as the target location the following property needs to be set: ohi.incident.target=datafileset. With this option enabled, trace files are written to datafile sets having the following name pattern: _OHIIncidents_<yyyyMMdd>. Effectively, this means that trace files end up in a specific datafile set on a daily basis. This way trace files are accessible through the data file set integration point. It is possible to download individual trace files, or all the trace files (zipped) of a specific day (i.e. a specific trace file datafile set).

Datafile set trace files retention

When the option to use datafile set as the target location of trace files is activated, these datafile sets adhere to a retention regime. This means that a datafile set containing application trace files is automatically scavenged whenever a retention period expires. By default, datafile sets for application trace files are retained for 10 days. It is possible to configure this by specifying a number of days for the property: ohi.incident.datafileset.retentionperiod.

Identifying application trace files

Every application trace file has a reference to an incident ID. This incident ID is a system generated piece of information, uniquely identifying a specific incident. Incident IDs follow this pattern: [APP]-[C]-#[NR]-YYYYMMDD_HH24MMSS.

APP: the shortname for the Oracle Health Insurance Application
C: the operating context; U=User Interface, W=Web/REST service, P=Processing
NR: a sequence number
YYYYMMDD : the date of the incident
HH24MMSS : the time of the incident

This incident ID is typically used in end-user facing application elements, like the UI. This provides a visible cue to the end-user that something went wrong in the application, and that an incident has been written. As such, it can then further be used in working with a application operator to find out what is going on. It is quite possible that from the trace it is not clear what the actual cause of the problem is. In those specific cases typically an OHI Support ticket is logged, and the trace file is included.

The main OHI application logfile will also hold an ERROR log entry when a trace file is generated. An example of what this looks like when using the standard filesystem location as target is shown below:

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to /home/logs/ohi/datafile/dfl-23.0.x/ohi-datafile-integration-tests/target/trace/trace-DAT-W-#1-20201020_125100.log

In case the property ohi.incident.target is set to datafileset, this similar line would looks something like this:

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to https://foo.bar.invalid:9807/api/datafilesets/_OHIIncidents_20201020/datafiles/trace-DAT-W-%231-20201020_140358.log/data

Depending on the Oracle Health Insurance system component the information used in this ERROR log entry, might be reused for other purposes. As a canonical example: this information might be reused in a database log record attached to a specific work item.

Example trace file

To close this section an example of trace file is shown.

Some formatting applied to the below example for documentation purposes.

------------------------------------------------------------------------------------------------
 Application : DAT (1)
 Version     : 23.0.0-SNAPSHOT (2)
 Context     : WebService (3)
 Filename    : trace-DAT-W-#1-20201020_140951.log (4)
 Date        : October 20, 2020 (5)
 Time        : 2:09:51 pm (6)
-------------------------------------------------------------------------------------------------

 Unexpected error context: No additional context (7)

 java.lang.RuntimeException: Thrown for illustration purposes only (8)
	at ...IncidentDFWriterTest.generateTraceFile(IncidentDFWriterTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:532)
...
-------------------------------------------------------------------------------------------------

 ** THREAD DUMP: (9)

"InterruptTimer" Id=41 WAITING on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)

"oracle.jdbc....BlockReleaser" Id=40 TIMED_WAITING on BlockSource$ThreadedCaching@8fa4647
	at java.lang.Object.wait(Native Method)
	-  waiting on oracle.BlockSource$ThreadedCachingBlockSource$BlockReleaser@8fa4647
	at oracle.jdbc.driver.BlockSource$ThreadedCaching.run(BlockSource.java:329)

"Timer-0" Id=39 WAITING on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)
...
-------------------------------------------------------------------------------------------------
[END] (10)

1	Shortname of the application.
2	Full version of the application.
3	Operating context at time of the incident.
4	The trace filename.
5	The date of the incident.
6	The time of the incident.
7	Optional, additional context of the incident.
8	The stacktrace of the component from the incident originated.
9	A thread dump of all the other threads that were operating at the time of the incident.
10	An indication that this is the last line of the trace file.