Dgraph request log and stdout/sterr log

Any Dgraph node creates two logs.

You can use these Dgraph logs to troubleshoot queries, or to track performance of particular queries or updates. The Dgraph logs are text files.

Collecting debugging information for the Dgraph

Before attempting to debug an issue with the Dgraph, collect the following information:
  • A zip file containing the Dgraph request log and stdout/sterr log for the Dgraph instances.
  • Hardware specifications and configuration.
  • Description of the BDD cluster deployment. How many nodes are hosting Dgraph instances in your BDD deployment? Are Dgraph instances hosted on nodes that also host other components of BDD, or are the hosting machines dedicated to hosting only the Dgraph instances?
  • The data from the Dgraph Statistics page. See About Dgraph statistics.
  • Description of which Dgraph instance is affected.

Dgraph request log

The Dgraph request log (also called the query log) contains one entry for each request processed. The requests are sorted by their timestamp.

The request log name and storage location is specified by the Dgraph --log flag. By default, the name and location of the log file is set to: 
$BDD_HOME/dgraph/bin/dgraph.reqlog
The format of the Dgraph request log consists of fourteen fields, which contain the following information:
  • Field 1: Timestamp (UNIX Time with milliseconds).
  • Field 2: Client IP Address.
  • Field 3: Outer Transaction ID, if defined.
  • Field 4: Request ID.
  • Field 5: Response Size (bytes).
  • Field 6: Total Time (fractional milliseconds).
  • Field 7: Processing Time (fractional milliseconds).
  • Field 8: HTTP Response Code (0 on client disconnect).
  • Field 9: - (unused).
  • Field 10: Queue Status. On request arrival, the number of requests in queue (if positive) or the number of available slots at the same priority (if negative).
  • Field 11: Thread ID.
  • Field 12: HTTP URL (URL encoded).
  • Field 13: HTTP POST Body (URL encoded; truncated to 64KBytes, by default; - if empty).
  • Field 14: HTTP Headers (URL encoded).

Note that a dash (-) is entered for any field for which information is not available or pertinent.

By default, the Dgraph truncates the contents of the body for POST requests at 64K. This default setting saves disk space in the log, especially during the process of adding large numbers of records to the data domain. If you need to review the log for the full contents of the POST request body, contact Oracle support.

Dgraph stdout/stderr log

The Dgraph redirects its stdout/stderr output to the log file specified by the Dgraph --out flag. By default, the name and location of the file is: 
$BDD_HOME/logs/dgraph.out

You can specify a new log location by changing the DGRAPH_OUT_FILE parameter in the bdd.conf file and then restarting the Dgraph.

The Dgraph stdout/stderr log includes startup messages as well as warning and error messages. You can increase the verbosity of the log via the Dgraph -v flag. You can also set the logging variables to toggle logging verbosity for specified features, which are described in Configuring verbose logging for a Dgraph target.

Note that the Dgraph stdout/stderr log reports startup and shutdown times (and other informational messages) using the system's local time zone, with no zone label displayed, but displays warning and error messages in UTC.

Using grep on the Dgraph request log

When diagnosing performance issues, you can use grep with a distinctive string to find individual requests in the Dgraph request log. For example, you can use the string: 
value%3D%22RefreshDate
If you have Studio, it is more useful to find the X-Endeca-Portlet-Id HTTP Header for the portlet sending the request, and grep for that. This is something like: 
X-Endeca-Portlet-Id: endecaresultslistportlet_WAR_endecaresultslistportlet_INSTANCE_5RKp_LAYOUT_11601
As an example, if you set: 
PORTLET=endecaresultslistportlet_WAR_endecaresultslistportlet_INSTANCE_5RKp_LAYOUT_11601
then you can look at the times and response codes for the last ten requests from that portlet with a command such as: 
grep $PORTLET Discovery.reqlog | tail -10 | cut -d ' ' -f 6,7,8

The command produces output similar to: 

20.61 20.04 200
80.24 79.43 200
19.87 18.06 200
79.97 79.24 200
35.18 24.36 200
87.52 86.74 200
26.65 21.52 200
81.64 80.89 200
28.47 17.66 200
82.29 81.53 200
There are some other HTTP headers that can help tie requests together:
  • X-Endeca-Portlet-Id — The unique ID of the portlet in the application.
  • X-Endeca-Session-Id — The ID of the user session.
  • X-Endeca-Gesture-Id — The ID of the end-user action (not filled in unless Studio has CLIENT logging enabled).
  • X-Endeca-Request-Id — If multiple dgraph requests are sent for a single Dgraph Gateway request, they will all have the same X-Endeca-Request-Id.