Viewing System Log Data

Logs are collected from all over the system and aggregated in Loki.

Using Grafana Explore Queries

System log data can be queried, filtered, and displayed using Grafana Explore queries.

Loki Logs

Loki uses labels to categorize log messages. A query specifies labels, and Loki displays the service and application log messages that match the query selections.

Labels are key-value pairs. Use the following procedure to select labels for your query.

  1. Open the Grafana home page.

  2. Open the Explore pane.

    In the vertical menu bar on the left side of the page, click Explore (the compass icon).

  3. To query Loki data, select Loki from the Explore data source menu at the top of the page to the right of the "Explore" title.

    Loki query options are displayed. For example, a Log Browser menu is shown at the top of the page.

  4. Query and filter the logs.

    The following methods are similar. Both methods allow you to select labels and values from lists. The second method enables you to more easily select multiple labels and multiple values for one query.

    Once you have created a query, you can select the same query again from the history list.

Additional query options:

  • Add query. Click the Add query button to create another query and show the result of all separate queries together in the same timeline and message list.

  • Query history. Run a query that was previously run, or copy or delete the query, add a comment to the query, or star the query so that you can use the Starred button to list only starred queries. At the top of the Query history list you can enter a search string to filter the list, and you can select how to order the list.

  • Recurring run. Click the arrow on the Run query button, and select an interval from the menu. To stop the recurring runs, select Off at the top of the menu.

The timeline is displayed below the Log browser section of the Explore pane. Below the timeline, the log messages that match the query are displayed.

Messages are color-coded both in the timeline and in the message list to indicate whether the message is informational, a warning, error, or other.

Use the Query type button to choose to show the results over a range of time or at just one point in time. Use the range button at the top of the page (see the clock icon) to set the range.

Select a portion of the timeline to zoom in to focus on a smaller amount of data. To zoom out, use the magnifying glass button at the top of the page next to the range button.

In the message list, click the arrow on the left side of the time stamp of a message to display all labels that match that message. You can then click the plus + magnifying glass icon to add that label to your query results or click the minus - magnifying glass icon to remove that label from your query results. Notice that the query that you entered changes.

Enter a Query in the Text Field

  1. In the text field to the right of the Log browser button, enter the open brace { character.

    The closed brace is automatically added, and a list of labels pops up.

  2. Select a label from the list.

    You might need to scroll the list to see all labels, or you can start typing a label name to filter the list.

    The selected label is inserted into the query in the text field, an equals sign is added, and a list of values for that label pops up.

  3. Select a value from the list.

    You might need to scroll the list to see all values, or you can start typing a value name to filter the list.

    The selected value is inserted inside quotation marks.

  4. If you want to further filter the query result, enter a comma.

    The list of labels pops up again, followed by the list of values after you select a label.

  5. Run the query.

    Type Shift+Enter, or click the Run query button in the upper right corner of the pane.

    The timeline and log messages are displayed below the query building options.

More Easily Build a Complex Query

Click the Log browser button so that the arrow on the button points down.

A query builder is displayed with the following steps:

  1. Select labels.

    Step 1 displays a row of buttons with a label name on each button. When you click one of these label buttons, a list pops up under Step 2 that shows the values for that label.

    You can click more than one label button. If you click another label button, the list of values for the new label pops up with the first list of values under Step 2.

    When you click a label button that is already selected, that label is removed from the query.

  2. Choose values for the selected labels.

    Step 2 shows the list of values for each label that is selected in Step 1. You might need to scroll the list to see all possible values, or you can start typing a value name in the search field to filter all value lists.

    When you select a value from one list, some values might be removed from another list.

    You can select more than one value from a particular list. Selecting a value that is already selected removes that value from the query.

    As you select or deselect values, the query is built and displayed in Step 3.

  3. Show the query result.

    Click the Show Logs button in Step 3.

    The timeline and log messages are displayed below the query building options.

    The completed query is displayed in the field to the right of the Log browser button. You can edit the query in the Log browser field and click the Run query button to show a new result.

Audit Logs

The audit logs can be consulted as separate categories. From Log browser lists, you can select the following audit labels. As described in Loki Logs, either enter the queries shown in the following list in the text field, or select job or log from the Log labels list, and then select one of the values shown in the following list. See also the example custom query immediately following this list.

  • job="vault-audit"

    Use this log label to filter for the audit logs of the Vault cluster. Vault, a key component of the secret service, keeps a detailed log of all requests and responses. You can view every authenticated interaction with Vault, including errors. Because these logs contain sensitive information, many strings within requests and responses are hashed so that secrets are not shown in plain text in the audit logs.

  • job="kubernetes-audit"

    Use this log label to filter for the audit logs of the Kubernetes cluster. The Kubernetes audit policy is configured to log request metadata: requesting user, time stamp, resource, verb, etc. Request body and response body are not included in the audit logs.

  • job="audit"

    Use this log label to filter for the Oracle Linux kernel audit daemon logs. The kernel audit daemon (auditd) is the userspace component of the Linux Auditing System. It captures specific events such as system logins, account modifications, and sudo operations.

  • log="audit"

    Use this log label to filter for the audit logs of the ZFS Storage Appliance.

In addition to using the log labels from the list, you can also build custom queries. For example, to filter for the audit logs of the admin service and API service, enter the following query into the Log browser text field:

{job=~"(admin|api-server)"} | json tag="tag" | tag=~"(api-audit.log|audit.log)"

To execute, either type Shift+Enter, or click the Run query button in the upper right corner of the Explore pane.

LBaaS Logs

The Load Balancer as a Service (LBaaS) logs can be consulted as separate categories. From Log browser lists, you can select the following audit labels. As described in Loki Logs, either enter the queries shown in the following list in the text field, or select job or log from the Log labels list, and then select one of the values shown in the following list.

  • job="pca-lbctl"

    Use this log label to filter for the load balancer controller logs. You can view every client request that is being served. These logs contain API parameters and will contain error details when applicable.

  • job="pcalbmgr"

    Use this log label to filter for the load balancer instances (manager) logs. You can view every request that is being served. These logs primarily contain the load balancer's configuration and management.

In addition to using the log labels from the list, you can also build custom queries. For example, you can view the controller and manager logs together:

{job=~"pca-lbctl|pca-lbmgr"}

To execute, either type Shift+Enter, or click the Run query button in the upper right corner of the Explore pane.

Using the Vector Service

You can use the Vector service to send the information you want from Loki to an external location that you specify.

Beginning with Private Cloud Appliance Release 3.0.2-b1261765, Vector is installed, configured, and enabled on the appliance by default.

To specify which data you want and where you want the data sent, log in to the currently active management node as the root user and customize the Vector configuration file.

The Vector configuration file is at the following location on the management nodes: 

/nfs/shared_storage/log_streaming/pca_vector.yaml

Edit the configuration file to customize the sinks section. See the Vector Sinks reference. The following is a sample pca_vector.yaml file: 

# Copyright (c) 2024, Oracle and/or its affiliates.

# DO NOT TURN API OFF
# otherwise livenessProbe will fail
api:
  enabled: true
  # Bind to 0.0.0.0. Otherwise the API will not be exposed outside the container.
  address: "0.0.0.0:8686"

sources:
  fluentd_source:
    type: fluent
    address: "0.0.0.0:8080"
    mode: tcp
    encoding:
      codec: json

transforms:
  log_event:
    type: remap
    inputs:
      - fluentd_source
    source: |
      log(.)

sinks:
  loki_sink:
    type: loki
    inputs:
      - fluentd_source
    endpoint: http://your_external_location:3100
    encoding:
      codec: json
    labels:
      job: "vector"
      namespace: "default"
      system: "pca_name.example.com"
      filename: "{{tag}}"

In the endpoint value, your_external_location can be an IP address or a domain name. At this location, you could install Grafana or use other tools to filter, manipulate, and display the data.

The value of the filename label that is shown in the example causes the name of the source log file to be shown in the Vector Loki Sink output. You can then use that file name as a label to search within Loki and Grafana.

The following is an example Splunk sink:

sinks:
  splunk_sink:
    type: splunk_hec_logs
    inputs:
      - source_id
    endpoint: https://splunk_endpoint
    token: splunk-hec-token
    encoding:
      codec: json
    tls:
      ca_file: "/path/to/ca.pem"

The splunk-hec-token is required to send logs to Splunk. The ca_file is optional if you are using HTTPS. For more information about HTTP Event Collector and how to configure and use Splunk, see the Splunk documentation.

The following command reports the status of the log streaming pod (the Vector service): 

# kubectl get pods -n log-streaming
NAME                           READY   STATUS    RESTARTS   AGE
log-streamer-bc4d65d78-ndrsk   1/1     Running   0          14d

The following command prints the logs from the log-streamer-bc4d65d78-ndrsk pod, which has only one container. For more information about the kubectl logs command, see kubectl logs on the kubernetes.io site. 

# kubectl logs log-streamer-bc4d65d78-ndrsk -n log-streaming