Introduction

This tutorial shows you how to use Visual Builder to build a custom application interface using OCI Logging Analytics as a backend data source. For this tutorial, we will use OCI Audit Logs as an example. However, this approach will work for any type of log data you have in Oracle Logging Analytics. Creating your own white label application interface would be ideal when you want to separate the cloud infrastructure from your customer's view and have an independent visualization of the analytics.

The application is built using Visual Builder to connect to Oracle Logging Analytics in your tenancy. The analysis data is retrieved from Oracle Logging Analytics log group using REST API. In Oracle Logging Analytics, you must first ingest OCI Audit Logs by creating a service connector, review the aggregated data in a dashboard and explore the available logs in the Log Explorer.

The following are examples of JET visualizations created using Visual Builder:

Description of JET_visualization_1.png follows
Description of the illustration JET_visualization_1
Description of JET_visualization_2.png follows
Description of the illustration JET_visualization_2

Oracle Visual Builder is a cloud-based software development Platform as a Service (PaaS) and a hosted environment for your application development infrastructure. It provides an open-source standards-based solution to develop, collaborate on, and deploy applications within Oracle Cloud.

Description of vbcs-architecture.png follows
Description of the illustration vbcs-architecture

Note:

The steps presented in this tutorial are primarily for using Visual Builder to create an application interface to customize your view of the analysis of OCI Audit Logs. However, you can create the interface for any other type of logs which are ingested using any ingestion method supported by Oracle Logging Analytics.

Using the steps in this tutorial, you can access data cross-tenancy where your application runs in one tenancy and makes REST API calls to Oracle Logging Analytics in a different OCI tenancy.

Objectives

What you will accomplish upon completing this tutorial.

  • Ingest OCI Audit Logs using Oracle Logging Analytics
  • Set up Visual Builder to create an application interface
  • Create an application interface for viewing the analyzed log data

Prerequisites

  • Enable Oracle Visual Builder. See How to Begin with Oracle Visual Builder Subscriptions.
  • Set up users and groups, and create IAM policies for using Oracle Visual Builder. See Setting Up Users and Groups.
  • You can provide access control to your logs by creating policies that restrict access to the log group where the logs are located. If you want to provide access control to multiple sets of logs, then create separate compartments and log groups for each set of logs, and create policies for each of those compartments.

Task 1: Ingest Logs into Logging Analytics

For step-by-step instructions to set up ingestion for OCI Audit Logs, see Oracle Cloud Infrastructure Logging Analytics Quick Start Guide. If you are yet to onboard Logging Analytics, then the instructions are available in the same link to first onboard the service.

After the ingestion is set up, verify that the logs are collected and can be viewed in the Log Explorer. This ensures that the log data is available in Oracle Logging Analytics for analysis. Next, create the application using Visual Builder to view the analysis of the log data.

Task 2: Access Visual Builder and Build Your Visual Application

  1. Access Oracle Visual Builder.

  2. Develop Your Application.

    The dashboard application provides multiple views of related data from the same domain in single page. Visual Builder fragments are ideal to implement such reusable user interface widgets. The application pages will be a collection of the fragment widgets arranged in a responsive format to work for desktop as well as mobile screen size. See Work with Fragments.

    Access the example application created by Oracle and use it as a starting point to customize it to your use case. See Sample Code Provided by Oracle on Github.

    Some important details for implementing the application:

    • Widgets: The dashboard page will have a filtering user interface applicable to all widgets like selecting the time frame, compartment, etc. The widgets receive these filter criteria through fragment input parameters.
    • Fragments: The code for the fragment widgets define the input parameters, which of those parameters are the required ones, and which of them are optional. The fragment listens on changes of its input parameters and refreshes data that it presents automatically upon any change by fetching the updated data according to the new filtering constraints.

      Fragment has the vbEnter event which when fired starts to run the query for Oracle Logging Analytics and stores the results into the ADP variable for the presentation layer.The listener on the fragment input parameters re-runs the same logic as vbEnter event whenever the input parameters change.

      A widget can allow interaction with the data, for example, click a chart segment, select a table row, etc. Such events are fired from the fragment using the fragment custom event to allow the parent dashboard page to respond to them in a suitable way, for example, by drilling down in a different dashboard page, by filtering existing data, etc.

      The fragment variables and input parameters are defined in the file log-records-trend-fragment.json in Oracle custom data format. Explore the fragments in Design Time for a friendly interface and update the properties. See Customize How a Fragment Variable is Displayed in the Properties Pane.

      Fragment Example for Log Records Trend Chart: log-records-trend-fragment.html

      Fragment Example for Data Loading Chain: fetchRecordsAsync.js


    • Page user interface: The page user interface presents data from ArrayDataProvider variable and typically is a chart, table, or anything suitable. What query is executed and how the resulting data are presented is the private implementation of the fragment. This can be a table component, pie chart, bar chart, summary label, etc.

  3. Calling REST API:

    This is implemented in AppModule.fetchData function using Rest.get() call. This method will implement data access layer. The details are discussed in the following sections.

Task 3: Create REST-based Call to Query Logging Analytics Data

Oracle Logging Analytics REST API allows to query analytics data. The same REST call is used to query different types of data by specifying the query in the request body of the REST call. The REST call has several input parameters but this application will use only a few of them. These parameters are specified in a JSON file which is passed as the body in the call. The REST API endpoint URL is of the format:

https://loganalytics.<region>.oci.oraclecloud.com/20200601/namespaces/<namespaceName>/search/actions/query

To get the value of namespaceName, under Oracle Logging Analytics, go to Administration, and go to Service Details.

The JSON payload for the POST method is of the following format:

{
      "subSystem": "LOG",
      "queryString": "'Log Source' = 'OCI Audit Logs' | timestats count as logrecords by Event | sort -logrecords",
      "shouldRunAsync": true,
      "shouldIncludeTotalCount": true,
      "compartmentId": "ocid1.tenancy.oc1..someID",
      "compartmentIdInSubtree": true,
      "scopeFilters": [
      ],
      "timeFilter": {
            "timeStart": "2023-05-23T18:39:16.206Z",
            "timeEnd": "2023-05-24T02:39:16.206Z",
            "timeZone": "America/Los_Angeles"
      },
      "maxTotalCount": 2000
}

In the above JSON payload, the parameters are:

  • queryString: Query to fetch the data, and to aggregate them. For more details on how to write queries and their syntax, see Query Search and Command Reference.
  • compartmentId: OCID of compartment where the logs are stored
  • timeFilter: The start time, end time, and the time zone of the log data to query

For details about the Query API call, see API Reference and Endpoints: Query.

The POST REST API call returns a JSON response which contains the response header:

opc-work-request-id: ocid1.loganalyticsqueryjobworkrequest.oc1.region.someID

This identifies a job on the OCI side to collect the data and return them. To query the results of that job, use the same REST endpoint in the GET method with the workRequestId URL query parameter pointing to opc-work-request-id header value. For example, the JSON payload to query the result of the job: 

{
    "workRequestId": "ocid1.loganalyticsqueryjobworkrequest.oc1.region.someID",
    "limit": "1000",
    "shouldIncludeColumns": false
}

It may take some time till query response is ready. When you call the GET method, Oracle Logging Analytics responds with the results available at that time. It may be partial which is why the percentComplete is important. Check that percentComplete in the JSON response is equal to 100. Your application may decide to start rendering partial results before the rest of the results are available. Repeat the REST call after a short delay to get the next batch of results if the processing is not 100% complete, if necessary.

Task 4: Call Logging Analytics REST API from Visual Builder

The Rest.get() call is implemented in the AppModule.fetchData function in the file app-flow.js. This method implements the data access layer.

In the following example code from the function AppModule.fetchData, the POST and GET methods are implemented:

async fetchData_OCI_API_Signature(compartmentId, timeFilter, query) {
      const postQueryResult = await Rest.get("oci-signature/postQuery").body(
        {
          "subSystem": "LOG",
          "queryString": query,
          "shouldRunAsync": true,
          "shouldIncludeTotalCount": true,
          "compartmentId": compartmentId,
          "compartmentIdInSubtree": true,
          "scopeFilters": [
          ],
          "timeFilter": timeFilter,
          "maxTotalCount": 2000
        }
      ).fetch();

      if (postQueryResult.response.status !== 201) {
        return { error: true, message: JSON.stringify(postQueryResult.body) };
      }

      const workRequestId = postQueryResult.response.headers.get("Opc-Work-Request-Id");

      const delay = ms => new Promise(res => {setTimeout(res, ms);});
      let retries = 6;
      while (retries > 0) {
        retries--;
        const data = await Rest.get("oci-signature/getQuery").parameters(
          {
            workRequestId: workRequestId,
          }
        ).fetch();


        if (data.response.status !== 200) {
          return { error: true, message: JSON.stringify(postQueryResult.body) };
        } else if (data.body.percentComplete === 100) {
          return { error: false, body: data.body };
        }
        //console.log("response is not complete: body.percentComplete="+data.body.percentComplete);
        await delay(666);
      }
      return { error: true, message: "response not received in time" };

    }

Oracle Logging Analytics REST API is called using the same account that was used to setup the Service Connection between Visual Builder and Oracle Logging Analytics.

In order to call REST APIs from the Visual Builder application, the endpoints must be registered in Visual Builder through Service Connection and authentication must be provided. See Work with Services.

OCI supports signature based authentication which can also be used in Visual Builder. See Request Signatures and How Does Authentication with Fixed Credentials Work?.

The REST API endpoint URL that was discussed in the earlier section must be added as two endpoints, one for POST call and the other for GET call. The POST call must have the request header content-type set to application/json and request body set to JSON. The GET call must have the dynamic query parameter workRequestId of the type String.

Task 5: View Your Log Data in the New Visual Application

The dashboard page contains fragments in a layout which is responsive from desktop and mobile screen sizes. The page has the user interface component which allows users to narrow down the search criteria using start and end time for which the queries should be run. These filtering parameters are passed into individual fragments using the fragment input parameters.

The following are some of the widgets implemented by using the queries in the POST JSON body:

  1. Log Records Trend

    Regular bar chart presenting log record counts over time.

    'Log Source' = 'OCI Audit Logs' | timestats count as logrecords by 'Log Source' | sort -logrecords
  2. Top Log Labels

    Table listing top log labels and number of records. Label can be clicked and fragment will fire event that user wants to drill down into that label.

    Label != null | stats count as Records by Label | sort -Records | head limit = 5
  3. Count

    For given query and label the widget will execute that query and will present resulting total with the label. Number can be optionally clickable to allow user drill down and fragment will fire a custom even in such case.

    Active users:

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', 'oci-optimizer', scanplatform) | stats distinctcount('User Name') as 'Active Users'

    Total audit records: 

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', cloudguard, 'oci-optimizer', scanplatform, taggingcontrolplaneservice) | stats count
  4. Top Users

    Pie chart with first 10 most active users.

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', 'oci-optimizer', scanplatform, taggingcontrolplaneservice) | stats count by 'User Name' | head limit = 10
  5. Top actions

    Render in table event, method and their count.

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', cloudguard, 'oci-optimizer', scanplatform, taggingcontrolplaneservice, oke) and Path != 'null' | stats count as count by Method, Event | sort -count | top limit = 25 count

More Learning Resources