Application Trace Files

In case Oracle Health Insurance application runs into an unanticipated error, an application trace file generates.
Application trace files contain technical information that can assist in the root cause analysis of a problem.
Typically, you need to share application trace files when present with Oracle Health Insurance support with the product development team for analysis.

Controlling the Target Location for Trace Files

Filesystem Location

The application writes the trace files to the filesystem by default.
The ohi.incident.rootdir property governs the actual directory.
In case you did not set this property, the application writes the trace files to a /trace directory.
In that case, this is a directory that is relative to the location from where the application starts.
In case you set the ohi.incident.rootdir, the application uses the value of this property as the directory of trace files.
Note that the application creates this directory automatically, in case the directory does not exist.
The value for this property must have a directory that the O/S user that starts the application can write(and create) to.

Datafile Set Location

It is possible to have the system use datafile sets as the location of trace files.
This is especially useful in cloud-based deployments, where there is access restriction to the physical filesystem.
To have datafile set as the target location, you need to set the following property: ohi.incident.target=datafileset.
If you have enabled this option, the application writes the trace files to datafile sets having the following name pattern: _OHIIncidents_<yyyyMMdd>.
Effectively, this means that trace files end up in a specific datafile set on a daily basis.
This way trace files are accessible through the integration-points:http-integration-points/data-file-set-integration-point.adoc.
It is possible to download individual trace files, or all the trace files (zipped) of a specific day (that is, a specific trace file datafile set).

Datafile Set Trace Files Retention

If you activate the option to use a datafile set as the target location of trace files, these datafile sets adhere to a retention regime.
This means that a datafile set containing application trace files scavenges automatically whenever a retention period expires.
By default, the application keeps the datafile sets for application trace files for 10 days.
It is possible to configure this by specifying the number of days for the property: ohi.incident.datafileset.retentionperiod.

Identifying Application Trace Files

Every application trace file has a reference to an incident ID.
This incident ID is a system-generated piece of information, uniquely identifying a specific incident.
Incident IDs follow this pattern: [APP]-[C]-#[NR]-YYYYMMDD_HH24MMSS.

APP: the short name for the Oracle Health Insurance Application
C: the operating context; U=User Interface, W=Web/REST service, P=Processing
NR: a sequence number
YYYYMMDD : the date of the incident
HH24MMSS : the time of the incident

The end-user-facing application elements, like the UI, typically use this incident ID.
This provides a visible cue to the end-user that something went wrong in the application, and that the application writes an incident.
As such, you can use it further with an application operator to find out what is going on.
It is quite possible that from the trace, it is not clear what the actual cause of the problem is.
In those specific cases, typically Oracle Health Insurance application logs a support ticket, including the trace files.

The main Oracle Health Insurance application logfile will also hold an ERROR log entry when the application generates a trace file.
Following is an example of what this looks like when using the standard filesystem location as the target:

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to /home/logs/ohi/datafile/dfl-23.0.x/ohi-datafile-integration-tests/target/trace/trace-DAT-W-#1-20201020_125100.log

In case you set the property ohi.incident.target to datafileset, this similar line looks something like this:

We apply some formatting in the following example for documentation.

2020-10-20 12:51:00,249 [ main ] ERROR c.o.h.d.s.i.c.IncidentDFWriterTest - Incident: DAT-W-#1-20201020_125100 written to https://foo.bar.invalid:9807/api/datafilesets/_OHIIncidents_20201020/datafiles/trace-DAT-W-%231-20201020_140358.log/data

Depending on the Oracle Health Insurance application, you can reuse the information in the ERROR log entry for other purposes.
As a canonical example: you can reuse this information in a database log record attaching it to a specific work item.

Example Trace File

Following is an example of a trace file.

We apply some formatting in the following example for documentation.

------------------------------------------------------------------------------------------------
 Application : DAT (1)
 Version     : 23.0.0-SNAPSHOT (2)
 Context     : WebService (3)
 Filename    : trace-DAT-W-#1-20201020_140951.log (4)
 Date        : October 20, 2020 (5)
 Time        : 2:09:51 pm (6)
-------------------------------------------------------------------------------------------------

 Unexpected error context: No additional context (7)

 java.lang.RuntimeException: Thrown for illustration purposes only (8)
	at ...IncidentDFWriterTest.generateTraceFile(IncidentDFWriterTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:532)
...
-------------------------------------------------------------------------------------------------

 ** THREAD DUMP: (9)

"InterruptTimer" Id=41 WAITING on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@7c689379
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)

"oracle.jdbc....BlockReleaser" Id=40 TIMED_WAITING on BlockSource$ThreadedCaching@8fa4647
	at java.lang.Object.wait(Native Method)
	-  waiting on oracle.BlockSource$ThreadedCachingBlockSource$BlockReleaser@8fa4647
	at oracle.jdbc.driver.BlockSource$ThreadedCaching.run(BlockSource.java:329)

"Timer-0" Id=39 WAITING on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@53a2dbf2
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)
...
-------------------------------------------------------------------------------------------------
[END] (10)

1	Shortname of the application.
2	Full version of the application.
3	Operating context at time of the incident.
4	The trace filename.
5	The date of the incident.
6	The time of the incident.
7	Optional, additional context of the incident.
8	The stacktrace of the component from the incident originated.
9	A thread dump of all the other threads that were operating at the time of the incident.
10	An indication that this is the last line of the trace file.