Using Grafana

With Grafana, Oracle Private Cloud Appliance offers administrators a single, visual interface to the logs and metrics collected at all levels and across all components of the system.

This section provides basic guidelines to access Grafana and navigate through the logs and monitoring dashboards. For additional information about Grafana services and how to use Grafana, see the Oracle Systems blog Oracle PCA X9-2 Monitoring and Alerting with Grafana.

The Grafana Home Page

Do one of the following to access Grafana:

Service Enclave admin user.
1. Log in to the Service Web UI.
2. On the right side of the dashboard, click the Monitoring tile.
  
  The Grafana login page opens in a new browser tab. In the future, you can go directly to this Grafana login page as described in "Any Grafana user" below.
3. Enter your user name and password at the prompts.
  
  You can create new users, and give the new users the direct URL to the Grafana login page. To create new Grafana users, see Adding Grafana Users. For the direct URL to the Grafana login page, see "Any Grafana user" below.
Any Grafana user.

You do not need to be a Service Enclave user.
1. Go to the Grafana login page.
  
  The Grafana login page is https://grafana.pca_name.your_domain/login, where pca_name is the name of the Private Cloud Appliance.
2. Enter your user name and password at the prompts.

The Welcome panel on the Grafana home page contains many links to grafana.com for information about how to use Grafana, such as how to create your own dashboards, queries, and alerts. You can also find Grafana tutorials on Oracle Private Cloud Appliance 3.x on the Oracle Learning YouTube channel or search for Grafana on Oracle Blogs.

On the left side of the home page is a vertical bar with icons that open the list of dashboards or the list of alerts, for example, or provide access to system logs as described in Accessing System Logs. Your user icon near the bottom of the bar enables you to change your preferences settings or log out. The Grafana logo at the top of the bar takes you back to the Grafana home page.

The Grafana Time Line

When logs and metrics are stored in Prometheus, they are given a time stamp based on the time and time zone settings of the Private Cloud Appliance. However, Grafana displays the time based on user preferences, which might result in an offset because you are in a different time zone.

Use the following instructions if you want to synchronize the time line in the Grafana visualizations with the time zone of the appliance:

Near the bottom of the vertical menu bar on the left side of the Grafana page, click your user account icon and click the Preferences option on the submenu that pops up.
In the Preferences section of the page, change the Timezone setting to the same time zone as the appliance.
Click the Save button at the bottom of that section to apply the change.

Monitoring Multiple Private Cloud Appliance X9-2 Systems

If you need to deploy an external Grafana service with variable-driven dashboards to monitor multiple Oracle Private Cloud Appliance X9-2 systems, see the following resources:

Adding Grafana Users

This section describes adding users and teams of users and granting permissions to use folders and dashboards.

To add a new user, perform the following procedure as the admin user:

In the vertical menu bar on the left side of the Grafana home page, click the Server Admin (shield) icon.
On the Server Admin drop down menu, click Users.
Click the New User button.
Enter the requested information, and click the Create User button.

By default, the new user has the Viewer role. You could modify the user to change the role. Another way to change a user's access is to add the user to a team that has the required access.

The following are the Grafana user roles:

Admin: Has access to all organization resources, including dashboards, users, and teams.

Editor: Can view and edit dashboards, folders, and playlists.

Viewer: Can view dashboards and playlists.

By default, an editor can edit all of the listed resources, and a viewer can view all of the listed resources. A user with the Admin role can grant or restrict permissions to specific resources for specific roles, teams, and users. For example, click the Permissions tab on a folder to change the permissions to that folder for the Editor or Viewer roles. Click the Add Permissions button on the Permissions tab to add permissions for specific users or teams.

To create a new team, perform the following procedure as the admin user:

In the vertical menu bar on the left side of the Grafana home page, click the Configuration (gear) icon.
On the Configuration drop down menu, click Teams.
Click the New Team button.
Enter the requested information, and click the Create button.
Click the Add Member button.
In the Add team member box, click the drop-down arrow and select the user you want to add to the team.
Click the Add to team button.
Click the Settings tab at the top of the page to modify team settings such as home dashboard and time zone.

Use folders to grant permissions to users and teams. Perform the following procedure as the admin user:

In the vertical menu bar on the left side of the Grafana home page, click the Dashboards (grid) icon.
On the Dashboards drop down menu, click Manage.
For the folder for which you want to grant teams and users permissions, click Go to folder.
At the top of the folder page, click the Permissions tab.
Click the Add Permission button.
In the Add Permission For box, select the team or user, and select the role for the user or for all users on the team.
Click the Save button.

You can also grant permissions for specific dashboards in a folder. Perform the following procedure as the admin user:

In the vertical menu bar on the left side of the Grafana home page, click the Dashboards (grid) icon.
On the Dashboards drop down menu, click Manage.
Click the name of the folder that contains the dashboard, and then click the dashboard.
At the top of the dashboard page, click the gear icon.
In the menu on the left side of the page, click Permissions.
Click the Add Permission button.
In the Add Permission For box, select the team or user, and select the role for the user or for all users on the team.
Click the Save button.
Click the Save Dashboard button.

Using Grafana Dashboards

Oracle provides a number of predefined Grafana dashboards organized into folders. Use any of the following to display the list of folders of dashboards:

The magnifying glass icon in the vertical menu bar on the left side the page
The Dashboards > Manage option in the menu bar
The dashboards Home button to the right of the Grafana logo at the top of the menu bar

Click a folder name or the arrow to the right of a folder name to show the dashboards in that folder.

Use the buttons at the top of the list to toggle between showing the list of folders and showing the list of all dashboards.

In the search field at the top of the page, enter text from the name of a folder or dashboard to show only those dashboards.

Click the name of a dashboard to show the content of that dashboard. On a dashboard, you can click the star to the right of the dashboard name at the top of the page to list this dashboard on the Grafana home page for faster access.

The dashboard shows information such as the query, graphs of the data collected over time, and alerts set for that data.

You are able to modify most dashboards, but note that Oracle Support might require that information. The Grafana home page contains links to information for how to create your own dashboards and queries rather than modify dashboards that were provided by Oracle. For your custom dashboards, first create one or more folders to keep these new dashboards separate from dashboards provided by Oracle.

Using Grafana Alerts

Oracle provides a predefined a set of alerts. You can also add your own alerts. You can show only alerts that are in a specified state, such as Alerting. You can display detailed information about the alert, including the values that trigger the alert and that trigger a state change. In many cases, you can change these values.

If an alert is in the Alerting state, view the alert definition to determine what caused the alert to go to that state, and then use this information to evaluate the component that the alert is monitoring and determine what action might be needed.

Browse Grafana Alerts

To view all alerts, click the bell icon in the vertical menu bar on the left side the page. An icon shows the status of each alert, and text below the alert name shows how long the alert has been in that status.

Enter text in the search field at the top of the list to show only alerts with that text in their names. Use the States list to show only alerts that are in the selected state: OK, Not OK, Alerting, No Data, Paused, Pending.

Use the "How to add an alert" button above the alerts list to create a new alert, or use information referenced on the Grafana home page to add or modify an alert, add a notification channel, and add a notification for a particular alert.

Click an alert name to see detailed information about the alert. This is the same page you see if you go to the dashboard, scroll to the metric, click the metric name, and select Edit.

Hover over the graph to list all data that is being monitored, for example each host, switch, device, or endpoint.

On the Alert tab below the graph, you can view and edit the rule. An alert rule consists of one or more queries and expressions, a condition, the frequency of evaluation, and optionally, the duration over which the condition is met. You can see how the alert state is set for various error conditions. You can send a notification message for this alert.

The state history button shows the last 50 state changes for this alert. Another button enables you to test the alert.

Add or Configure Notification Channels

To add or configure notification channels, click the bell icon in the menu bar on the left side the page and then select the Notification channels option, or select the Notification channels tab at the top of the list of alerts.

To change the configuration of an existing notification channel, click the name of the channel. When you are finished making changes, click the Save button. Click the Test button to send a test notification.

To add a notification channel, go to the Notification channels tab, click the New channel button, and fill out the page. Click the Save button. Click the Test button to send a test notification. Click the Back button to cancel and not create a new notification channel.

Configure Custom External Email Notifications

To configure email notification, open a service request (SR) for Oracle Support to do the initial configuration. When the initial configuration is complete, go to the Grafana alerts page, click the Notification channels tab, click the New channel button, and fill out the page, selecting Email in the Type field.

Configure Custom External HTTP/HTTPS Notifications

To configure external HTTP or HTTPS based custom alerts, you must first configure the proxy for Grafana as shown in the following example.

$ sudo curl -u admin_user_name -XPUT \
'https://api.PCA_system_name.your_domain/v1/grafana/proxy/config?http-proxy=proxy_fqdn:proxy_port&https-proxy=proxy_fqdn:proxy_port'
Enter host password for user 'admin_user_name':
Grafana proxy config successfully updated!

The Grafana pod is restarted. Run the following command until you see that the Grafana pod (sauron-sauron-grafana-unique_ID) is running:

$ kubectl get pods -n sauron

Checking the Health and Status of Hardware and Platform Components

The hardware and platform layers form the foundations of the system architecture. Any unhealthy condition at this level is expected to have an adverse effect on operations in the infrastructure services. A number of predefined Grafana dashboards allow you to check the status of those essential low-level components, and see the real-time and historic details of the relevant metrics.

The dashboards described in this section provide a good starting point for basic system health checks, and for troubleshooting if issues are found. You might prefer to use different dashboards, metrics, and visualizations instead. The necessary data, collected across the entire system, is stored in Prometheus, and can be queried and presented through Grafana in many different ways.

Grafana Folder	Dashboard	Description
Service Monitoring	Server Stats	This comprehensive dashboard displays telemetry data for the server nodes. It includes graphs for CPU and memory utilization, disk activity, network traffic, and so on. Some panels in this dashboard display a large number of time series in a single graph. Click to display a single time series, or hover over the graph to view detailed data at a specific time.
PCA 3.0 Service Advisor	Platform Health Check	This dashboard integrates the appliance health check mechanisms into the centralized approach that Grafana provides for logging and monitoring. By default, the Platform Health Check dashboard displays all health check services. Use the buttons above the Platform Health Check list to change the content of the list. Use the Platform Service list to select a single health checker. Use the Health Check Status list to display all results or only healthy results. Use the Filters list to select a filter and a value. Typically, if you see health check failures you want to start troubleshooting. For that purpose, each health check result contains a time stamp that serves as a direct link to the related Loki logs. To view the logs related to any health check result, click the time stamp.
My Dashboards (Read Only)	Node Exporter Full	This dashboard displays a large number of detailed metric panels for a single compute or management node. Use the Host button at the top of the page to display data for a different host. This dashboard could be considered a fine-grained extension of the Server Stats dashboard. The many different panels provide detailed coverage of the server node hardware status as well as the operating system services and processes. Information that you would typically collect at the command line of each physical node is combined into a single dashboard showing live data and its evolution over time. All dashboards in the My Dashboards folder provide data that would be critical in case a system-level failure needs to be resolved. Therefore, these dashboards cannot be modified or deleted.

Grafana Folder

Dashboard

Description

Service Monitoring

Server Stats

This comprehensive dashboard displays telemetry data for the server nodes. It includes graphs for CPU and memory utilization, disk activity, network traffic, and so on.

Some panels in this dashboard display a large number of time series in a single graph. Click to display a single time series, or hover over the graph to view detailed data at a specific time.

PCA 3.0 Service Advisor

Platform Health Check

This dashboard integrates the appliance health check mechanisms into the centralized approach that Grafana provides for logging and monitoring.

By default, the Platform Health Check dashboard displays all health check services. Use the buttons above the Platform Health Check list to change the content of the list. Use the Platform Service list to select a single health checker. Use the Health Check Status list to display all results or only healthy results. Use the Filters list to select a filter and a value.

Typically, if you see health check failures you want to start troubleshooting. For that purpose, each health check result contains a time stamp that serves as a direct link to the related Loki logs. To view the logs related to any health check result, click the time stamp.

My Dashboards (Read Only)

Node Exporter Full

This dashboard displays a large number of detailed metric panels for a single compute or management node. Use the Host button at the top of the page to display data for a different host.

This dashboard could be considered a fine-grained extension of the Server Stats dashboard. The many different panels provide detailed coverage of the server node hardware status as well as the operating system services and processes. Information that you would typically collect at the command line of each physical node is combined into a single dashboard showing live data and its evolution over time.

All dashboards in the My Dashboards folder provide data that would be critical in case a system-level failure needs to be resolved. Therefore, these dashboards cannot be modified or deleted.

Viewing and Interpreting Monitoring Data

The infrastructure services layer, which is built on top of the platform and enables all the cloud user and administrator functionality, can be monitored through an extensive collection of Grafana dashboards. These microservices are deployed across the three management nodes in Kubernetes containers, so their monitoring is largely based on Kubernetes node and pod metrics. The Kubernetes cluster also extends onto the compute nodes, where Kubernetes worker nodes collect vital additional data for system operation and monitoring.

The dashboards described in this section provide a good starting point for microservices health monitoring. You might prefer to use different dashboards, metrics and visualizations instead. The necessary data, collected across the entire system, is stored in Prometheus, and can be queried and presented through Grafana in many ways.

Grafana Folder	Dashboard	Description
Service Monitoring	ClusterLabs HA Cluster Details	This dashboard uses a bespoke Prometheus exporter to display data for HA clusters based on Pacemaker. On each HTTP request it locally inspects the cluster status, by parsing preexisting distributed data provided by the cluster components' tools. The monitoring data includes Pacemaker cluster summary, nodes and resource stats, and Corosync ring errors and quorum votes.
Service Monitoring	MySQL Cluster Exporter	This dashboard displays performance details for the MySQL database cluster. Data includes database service metrics such as uptime, connection statistics, table lock counts, as well as more general information about MySQL objects, connections, network traffic, memory and CPU usage, etc.
Service Monitoring	Service Level	This dashboard displays detailed information about RabbitMQ requests that are received by the fundamental appliance services. It allows you to monitor the number of requests, request latency, and any requests that caused an error.
Service Monitoring	VM Stats	This comprehensive dashboard displays resource consumption information across the compute instances in your environment. It includes graphs for CPU and memory utilization, disk activity, network traffic, and so on. The panels in this dashboard display a large number of time series in a single graph. You can click to display a single time series, or hover over the graph to view detailed data at a specific point on the time axis.
PCA 3.0 Service Advisor	Kube Endpoint	This dashboard focuses specifically on the Kubernetes endpoints and provides endpoint alerts. These alerts can be sent to a notification channel of your choice.
PCA 3.0 Service Advisor	Kube Ingress	This dashboard provides data about ingress traffic to the Kubernetes services and their pods. Two alerts are built-in and can be sent to a notification channel of your choice.
PCA 3.0 Service Advisor	Kube Node	This dashboard displays metric data for all the server nodes, meaning management and compute nodes, that belong to the Kubernetes cluster and host microservices pods. You can monitor pod count, CPU and memory usage, and so on. The metric panels display information for all nodes. In the graph-based panels you can click to view information for just a single node.
PCA 3.0 Service Advisor	Kube Pod	This dashboard displays metric data at the level of the microservices pods, allowing you to view the total number of pods overall and how they are distributed across the nodes. You can monitor their status per namespace and per service, and check if they have triggered any alerts.
PCA 3.0 Service Advisor	Kube Service	This dashboard displays metric data at the Kubernetes service level. The data can be filtered for specific services, but displays all by default. Two alerts are built-in and can be sent to a notification channel of your choice.
Kubernetes Monitoring Kubernetes Monitoring Containers Kubernetes Monitoring Node	(all)	These folders contain a large and diverse collection of dashboards with a wide range of monitoring data that covers most of the operations of the Private Cloud Appliance system Kubernetes cluster. For example, these metrics provide information about deployment, ingress, and usage of CPU, disk, memory, and network resources.
OKE Monitoring	CAPOCI	This dashboard shows metrics from the Cluster API Provider for OCI (CAPOCI), which is a component of Oracle Container Engine for Kubernetes (OKE). This dashboard monitors request status codes and response times for resources used by OKE such as compute instances and load balancers. The information about controller reconciliation is for Oracle Support.
OKE Monitoring	Cluster Time Monitoring	This dashboard shows the time taken for operations such as create or update a particular OKE cluster or node pool. Average time for these operations across all clusters and node pools also is shown.
OKE Monitoring	Metrics Meter	This dashboard shows the health of various targets used by the OKE service such as the Cluster API Provider (CAPI), the Cluster API Provider for OCI (CAPOCI), OKE, and prometheus-k8s.
OKE Monitoring	OKE Service	This dashboard shows the service level metrics for OKE. Examples of metrics on this dashboard include counts of requests such as cluster and node pool create, update, and delete, and counts of exception codes for various requests. The exception code counts help expose any patterns in request failures.

Monitoring System Capacity

It is important to track the key metrics that determine the system's capacity to host your compute instances and the storage they use. The detailed data for compute node load and storage usage can be found in the Grafana dashboards. Administrators also have direct access to the current consumption of CPU and memory as well as storage space.

Viewing CPU and Memory Usage By Fault Domain

These procedures display the number of compute nodes, the amount of total memory and free memory, and the number of total and free virtual CPUs for each fault domain.

The UNASSIGNED row refers to compute nodes that are not currently assigned to a fault domain. Because these compute nodes do not belong to a fault domain, their memory and CPU usage in a fault domain is zero.

To display this information and more for an individual compute node, select PCA Config > Rack Units from the navigation menu, or select the Rack Units tile on the Dashboard, and then click the name of a compute node in the list.

Using the Service Web UI

In the navigation menu, select PCA Config > Fault Domains.
Click the name of a fault domain to see this information for only that fault domain.

Using the Service CLI

Enter the getFaultDomainInfo command.

PCA-ADMIN> getFaultDomainInfo
Command: getFaultDomainInfo
Status: Success
Time: 2022-06-17 14:43:13,292 UTC
Data:
  id           totalCNs   totalMemory   freeMemory   totalvCPUs   freevCPUs
  --           --------   -----------   ----------   ----------   ---------
  UNASSIGNED   1          0.0           0.0          0            0
  FD1          2          1072.0        976.0        176          164
  FD2          1          984.0         984.0        120          120
  FD3          1          984.0         984.0        120          120

The Notes column is omitted from the above example.

Viewing Disk Space Usage on the ZFS Storage Appliance

The Service Enclave runs a storage monitoring tool called ZFS pool manager, which polls the ZFS Storage Appliance every 60 seconds. Using the Service CLI, you can display current information about the usage of available disk space in each ZFS pool. You can also set the usage threshold that triggers a fault when the threshold is exceeded.

Check the Storage Status of ZFS Pools

List ZFS pools.

PCA-ADMIN> list ZfsPool
Command: list ZfsPool
Status: Success
Time: 2022-10-10 08:44:11,938 UTC
Data:
  id                                     name
  --                                     ----
  e898b147-7cf0-4bd0-8b54-e32ec83d04cb   PCA_POOL
  c2f67943-df81-47a5-9713-06768318b623   PCA_POOL_HIGH

In a standard storage configuration, you only have one pool. If your system includes high-performance disk trays, then you can view usage information for each pool separately.

PCA-ADMIN> show ZfsPool id=e898b147-7cf0-4bd0-8b54-e32ec83d04cb
Command: show ZfsPool id=e898b147-7cf0-4bd0-8b54-e32ec83d04cb
Status: Success
Time: 2022-10-10 08:44:22,051 UTC
Data:
  Id = e898b147-7cf0-4bd0-8b54-e32ec83d04cb
  Type = ZfsPool
  Pool Status = Online
  Free Pool = 44879343128576
  Total Pool = 70506183131136
  Pool Usage Percent = 0.3634693989163486
  Name = PCA_POOL
  Work State = Normal

Configure the Fault Threshold of the ZFS Pool Manager

By default, the fault threshold is set to 80 percent full: usage percentage 0.8.

PCA-ADMIN> show ZfsPoolManager
Command: show ZfsPoolManager
Status: Success
Time: 2022-10-10 08:58:11,231 UTC
Data:
  Id = a6ca861b-f83a-4032-91c5-bc506394d0de
  Type = ZfsPoolManager
  LastRunTime = 2022-10-09 12:17:52,964 UTC
  Poll Interval (sec) = 60
  The minimum Zfs pool usage percentage to trigger a major fault = 0.8
  Manager's run state = Running

The following example sets the fault threshold to 75 percent full: usageMajorFaultPercent=0.75.

PCA-ADMIN> edit ZfsPoolManager usageMajorFaultPercent=0.75
Command: edit ZfsPoolManager usageMajorFaultPercent=0.75
Status: Success
Time: 2022-10-10 08:58:27,657 UTC
JobId: 67cfe180-f2a2-4d59-a676-01b3d73cffae

Accessing System Logs

Logs are collected from all over the system and aggregated in Loki. The log data can be queried, filtered, and displayed using Grafana.

Viewing Loki Logs

Loki uses labels to categorize log messages. A query specifies labels, and Loki displays the service and application log messages that match the query selections.

Labels are key-value pairs. Use the following procedure to select labels for your query.

Open the Grafana home page.
Open the Explore pane.

In the vertical menu bar on the left side of the page, click Explore (the compass icon).
To query Loki data, select Loki from the Explore data source menu at the top of the page to the right of the "Explore" title.

Loki query options are displayed. For example, a Log Browser menu is shown at the top of the page.
Query and filter the logs.

The following methods are similar. Both methods allow you to select labels and values from lists. The second method enables you to more easily select multiple labels and multiple values for one query.
- Enter a Query in the Text Field
- More Easily Build a Complex Query
Once you have created a query, you can select the same query again from the history list.

Additional query options:

Add query. Click the Add query button to create another query and show the result of all separate queries together in the same timeline and message list.
Query history. Run a query that was previously run, or copy or delete the query, add a comment to the query, or star the query so that you can use the Starred button to list only starred queries. At the top of the Query history list you can enter a search string to filter the list, and you can select how to order the list.
Recurring run. Click the arrow on the Run query button, and select an interval from the menu. To stop the recurring runs, select Off at the top of the menu.

The timeline is displayed below the Log browser section of the Explore pane. Below the timeline, the log messages that match the query are displayed.

Messages are color-coded both in the timeline and in the message list to indicate whether the message is informational, a warning, error, or other.

Use the Query type button to choose to show the results over a range of time or at just one point in time. Use the range button at the top of the page (see the clock icon) to set the range.

Select a portion of the timeline to zoom in to focus on a smaller amount of data. To zoom out, use the magnifying glass button at the top of the page next to the range button.

In the message list, click the arrow on the left side of the time stamp of a message to display all labels that match that message. You can then click the plus + magnifying glass icon to add that label to your query results or click the minus - magnifying glass icon to remove that label from your query results. Notice that the query that you entered changes.

Enter a Query in the Text Field

In the text field to the right of the Log browser button, enter the open brace { character.

The closed brace is automatically added, and a list of labels pops up.
Select a label from the list.

You might need to scroll the list to see all labels, or you can start typing a label name to filter the list.

The selected label is inserted into the query in the text field, an equals sign is added, and a list of values for that label pops up.
Select a value from the list.

You might need to scroll the list to see all values, or you can start typing a value name to filter the list.

The selected value is inserted inside quotation marks.
If you want to further filter the query result, enter a comma.

The list of labels pops up again, followed by the list of values after you select a label.
Run the query.

Type Shift+Enter, or click the Run query button in the upper right corner of the pane.

The timeline and log messages are displayed below the query building options.

More Easily Build a Complex Query

Click the Log browser button so that the arrow on the button points down.

A query builder is displayed with the following steps:

Select labels.

Step 1 displays a row of buttons with a label name on each button. When you click one of these label buttons, a list pops up under Step 2 that shows the values for that label.

You can click more than one label button. If you click another label button, the list of values for the new label pops up with the first list of values under Step 2.

When you click a label button that is already selected, that label is removed from the query.
Choose values for the selected labels.

Step 2 shows the list of values for each label that is selected in Step 1. You might need to scroll the list to see all possible values, or you can start typing a value name in the search field to filter all value lists.

When you select a value from one list, some values might be removed from another list.

You can select more than one value from a particular list. Selecting a value that is already selected removes that value from the query.

As you select or deselect values, the query is built and displayed in Step 3.
Show the query result.

Click the Show Logs button in Step 3.

The timeline and log messages are displayed below the query building options.

The completed query is displayed in the field to the right of the Log browser button. You can edit the query in the Log browser field and click the Run query button to show a new result.

Audit Logs

The audit logs can be consulted as separate categories. From Log browser lists, you can select the following audit labels. As described in Viewing Loki Logs, either enter the queries shown in the following list in the text field, or select job or log from the Log labels list, and then select one of the values shown in the following list. See also the example custom query immediately following this list.

job="vault-audit"

Use this log label to filter for the audit logs of the Vault cluster. Vault, a key component of the secret service, keeps a detailed log of all requests and responses. You can view every authenticated interaction with Vault, including errors. Because these logs contain sensitive information, many strings within requests and responses are hashed so that secrets are not shown in plain text in the audit logs.
job="kubernetes-audit"

Use this log label to filter for the audit logs of the Kubernetes cluster. The Kubernetes audit policy is configured to log request metadata: requesting user, time stamp, resource, verb, etc. Request body and response body are not included in the audit logs.
job="audit"

Use this log label to filter for the Oracle Linux kernel audit daemon logs. The kernel audit daemon (auditd) is the userspace component of the Linux Auditing System. It captures specific events such as system logins, account modifications, and sudo operations.
log="audit"

Use this log label to filter for the audit logs of the ZFS Storage Appliance.

In addition to using the log labels from the list, you can also build custom queries. For example, to filter for the audit logs of the admin service and API service, enter the following query into the Log browser text field:

{job=~"(admin|api-server)"} | json tag="tag" | tag=~"(api-audit.log|audit.log)"

To execute, either type Shift+Enter, or click the Run query button in the upper right corner of the Explore pane.

LBaaS Logs

The Load Balancer as a Service (LBaaS) logs can be consulted as separate categories. From Log browser lists, you can select the following audit labels. As described in Viewing Loki Logs, either enter the queries shown in the following list in the text field, or select job or log from the Log labels list, and then select one of the values shown in the following list.

job="pca-lbctl"

Use this log label to filter for the load balancer controller logs. You can view every client request that is being served. These logs contain API parameters and will contain error details when applicable.
job="pcalbmgr"

Use this log label to filter for the load balancer instances (manager) logs. You can view every request that is being served. These logs primarily contain the load balancer's configuration and management.

In addition to using the log labels from the list, you can also build custom queries. For example, you can view the controller and manager logs together:

{job=~"pca-lbctl|pca-lbmgr"}

To execute, either type Shift+Enter, or click the Run query button in the upper right corner of the Explore pane.