Introduction

This tutorial shows you how to use Visual Builder to build a custom application interface using OCI Logging Analytics as a backend data source. For this tutorial, we will use OCI Audit Logs as an example. However, this approach will work for any type of log data you have in Oracle Logging Analytics. Creating your own white label application interface would be ideal when you want to separate the cloud infrastructure from your customer's view and have an independent visualization of the analytics.

The application is built using Visual Builder to connect to Oracle Logging Analytics in your tenancy. The analysis data is retrieved from Oracle Logging Analytics log group using REST API. In Oracle Logging Analytics, you must first ingest OCI Audit Logs by creating a service connector, review the aggregated data in a dashboard and explore the available logs in the Log Explorer.

The following are examples of JET visualizations created using Visual Builder:

Description of JET_visualization_1.png follows
Description of the illustration JET_visualization_1
Description of JET_visualization_2.png follows
Description of the illustration JET_visualization_2

Oracle Visual Builder is a cloud-based software development Platform as a Service (PaaS) and a hosted environment for your application development infrastructure. It provides an open-source standards-based solution to develop, collaborate on, and deploy applications within Oracle Cloud.

Description of vbcs-architecture.png follows
Description of the illustration vbcs-architecture

Note:

The steps presented in this tutorial are primarily for using Visual Builder to create an application interface to customize your view of the analysis of OCI Audit Logs. However, you can create the interface for any other type of logs which are ingested using any ingestion method supported by Oracle Logging Analytics.

Using the steps in this tutorial, you can access data cross-tenancy where your application runs in one tenancy and makes REST API calls to Oracle Logging Analytics in a different OCI tenancy.

Objectives

What you will accomplish upon completing this tutorial.

  • Ingest OCI Audit Logs using Oracle Logging Analytics
  • Set up Visual Builder to create an application interface
  • Create an application interface for viewing the analyzed log data

Prerequisites

  • Enable Oracle Visual Builder. See How to Begin with Oracle Visual Builder Subscriptions.
  • Set up users and groups, and create IAM policies for using Oracle Visual Builder. See Setting Up Users and Groups.
  • You can provide access control to your logs by creating policies that restrict access to the log group where the logs are located. If you want to provide access control to multiple sets of logs, then create separate compartments and log groups for each set of logs, and create policies for each of those compartments.

Task 1: Ingest Logs into Logging Analytics

For step-by-step instructions to set up ingestion for OCI Audit Logs, see Oracle Cloud Infrastructure Logging Analytics Quick Start Guide. If you are yet to onboard Logging Analytics, then the instructions are available in the same link to first onboard the service.

After the ingestion is set up, verify that the logs are collected and can be viewed in the Log Explorer. This ensures that the log data is available in Oracle Logging Analytics for analysis. Next, create the application using Visual Builder to view the analysis of the log data.

Task 2: Note OCI Account Details

Note the following information from the tenancy where Oracle Logging Analytics is enabled:

  • Region
  • Tenancy name
  • Tenancy OCID
  • User OCID
  • User fingerprint key
  • Private key

    To download the keys, click the profile picture in the OCI console, click User Settings, under Resources click API Keys, and click Download Public and Private keys.

  • Service namespace

    To get the value of service namespace, under Oracle Logging Analytics, go to Administration, and go to Service Details.

Task 3: Download Sample Code Provided by Oracle and Update OCI Account Details

In order to call REST APIs from the Visual Builder application, the endpoints must be registered in Visual Builder through Service Connection and authentication must be provided. See Work with Services.

OCI supports signature based authentication which can also be used in Visual Builder. See Request Signatures and How Does Authentication with Fixed Credentials Work?.

  1. Access Oracle Visual Builder.

  2. Access the example application created by Oracle and use it as a starting point to customize it to your use case. See Sample Code Provided by Oracle on Github.

    Open it in Visual Builder to edit it.


  3. Edit the service connection of oci-signature:

    Description of oci-signature.png follows
    Description of the illustration oci-signature

  4. Edit server:
    Description of edit_server.png follows
    Description of the illustration edit_server
    • In the Instance URL, change the region to your region.
    • Change the tenancy name to your tenancy.
    • Edit Oracle Cloud Infrastructure API Signature:
      Description of edit_signature.png follows
      Description of the illustration edit_signature

      Enter the fingerprint key of the user who has access to Logging Analytics in the following format:

      <tenancy_ocid>/<user_ocid>/<fingerprint>

      Also, enter the private key that you downloaded earlier.


Task 4: Create REST-based Call to Query Logging Analytics Data

Oracle Logging Analytics REST API allows to query analytics data. The same REST call is used to query different types of data by specifying the query in the request body of the REST call. The REST call has several input parameters but this application will use only a few of them. These parameters are specified in a JSON file which is passed as the body in the call.

  1. In the Application, go to Dashboard, and click Variables.

    Under Variables, click CompartmentId.

    Change the Default Value in the right pane to your tenancy OCID. This ensures that the log search begins from the root compartment and continues further in the sub-compartments.

    Description of edit_variables.png follows
    Description of the illustration edit_variables

  2. Edit the POST method endpoint request payload:

    Go to POST method Source. Search for namespace. Replace it with the service namespace.

    Description of post_source.png follows
    Description of the illustration post_source

  3. Go to Endpoints Request Parameters, and verify the Path URL.

    The REST API endpoint URL is of the format:

    https://loganalytics.<region>.oci.oraclecloud.com/20200601/namespaces/<namespaceName>/search/actions/query

    Ensure that namespaceName is service namespace and the region is correct.

    Description of verify_endpoint.png follows
    Description of the illustration verify_endpoint

  4. Go to Endpoints Request Body.
    Description of request_body.png follows
    Description of the illustration request_body

    The example JSON payload for the POST method is of the following format:

    {
      "subSystem": "LOG",
      "queryString": "'Log Source' = 'OCI Audit Logs' | timestats count as logrecords by Event | sort -logrecords",
      "shouldRunAsync": true,
      "shouldIncludeTotalCount": true,
      "compartmentId": "ocid1.tenancy.oc1..someID",
      "compartmentIdInSubtree": true,
      "scopeFilters": [
      ],
      "timeFilter": {
            "timeStart": "2023-05-23T18:39:16.206Z",
            "timeEnd": "2023-05-24T02:39:16.206Z",
            "timeZone": "America/Los_Angeles"
      },
      "maxTotalCount": 2000
}

    In the above example, the parameters are:

    • queryString: Query to fetch the data, and to aggregate them. For more details on how to write queries and their syntax, see Query Search and Command Reference.
    • compartmentId: OCID of compartment where the logs are stored. Change the value of compartmentId to tenancy OCID. This ensures that the log search begins from the root compartment and continues further in the sub-compartments.
    • timeFilter: The start time, end time, and the time zone of the log data to query

    The POST call must have the request header content-type set to application/json and request body Media Type set to application/json.

    Save the example.


For details about the Query API call, see API Reference and Endpoints: Query.

The POST REST API call returns a JSON response which contains the response header:

opc-work-request-id: ocid1.loganalyticsqueryjobworkrequest.oc1.region.someID

This identifies a job on the OCI side to collect the data and return them. To query the results of that job, use the same REST endpoint in the GET method with the workRequestId URL query parameter pointing to opc-work-request-id header value. For example, the JSON payload to query the result of the job: 

{
    "workRequestId": "ocid1.loganalyticsqueryjobworkrequest.oc1.region.someID",
    "limit": "1000",
    "shouldIncludeColumns": false
}

Task 5: Test the REST API from Visual Builder

  1. Go to Endpoints Test. Click Send request.
    Description of endpoint_test.png follows
    Description of the illustration endpoint_test

    You can also directly edit the endpoint request body from Endpoints by clicking Test, under Request go to Body.

    Description of edit_body.png follows
    Description of the illustration edit_body

  2. Test the API and make sure that you get status 201.
    Description of api_test.png follows
    Description of the illustration api_test

The REST API endpoint URL that was discussed in the earlier section must be added as two endpoints, one for POST call and the other for GET call. The GET call must have the dynamic query parameter workRequestId of the type String.

It may take some time till query response is ready. When you call the GET method, Oracle Logging Analytics responds with the results available at that time. It may be partial which is why the percentComplete is important. Check that percentComplete in the JSON response is equal to 100. Your application may decide to start rendering partial results before the rest of the results are available. Repeat the REST call after a short delay to get the next batch of results if the processing is not 100% complete, if necessary.

Task 6: Test the Changes Made to the Application

Run the application to test the changes you made in the previous sections. This is an example application:

Description of JET_visualization_1.png follows
Description of the illustration JET_visualization_1

The dashboard page contains fragments in a layout which is responsive from desktop and mobile screen sizes. The page has the user interface component which allows users to narrow down the search criteria using start and end time for which the queries should be run. These filtering parameters are passed into individual fragments using the fragment input parameters.

The following are some of the widgets implemented by using the queries in the POST JSON body:

  1. Log Records Trend

    Regular bar chart presenting log record counts over time.

    'Log Source' = 'OCI Audit Logs' | timestats count as logrecords by 'Log Source' | sort -logrecords
  2. Top Log Labels

    Table listing top log labels and number of records. Label can be clicked and fragment will fire event that user wants to drill down into that label.

    Label != null | stats count as Records by Label | sort -Records | head limit = 5
  3. Count

    For given query and label the widget will execute that query and will present resulting total with the label. Number can be optionally clickable to allow user drill down and fragment will fire a custom even in such case.

    Active users:

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', 'oci-optimizer', scanplatform) | stats distinctcount('User Name') as 'Active Users'

    Total audit records: 

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', cloudguard, 'oci-optimizer', scanplatform, taggingcontrolplaneservice) | stats count
  4. Top Users

    Pie chart with first 10 most active users.

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', 'oci-optimizer', scanplatform, taggingcontrolplaneservice) | stats count by 'User Name' | head limit = 10
  5. Top actions

    Render in table event, method and their count.

    'Log Source' = 'OCI Audit Logs' and 'User Name' not in ('null', cloudguard, 'oci-optimizer', scanplatform, taggingcontrolplaneservice, oke) and Path != 'null' | stats count as count by Method, Event | sort -count | top limit = 25 count

Task 7: Customize Your Visual Application

You can edit the filters to add more filters in the javascript function used in the application:

Description of edit_filters.png follows
Description of the illustration edit_filters

For more detailed changes that you can make in the application, see Develop Your Application.

The dashboard application provides multiple views of related data from the same domain in single page. Visual Builder fragments are ideal to implement such reusable user interface widgets. The application pages will be a collection of the fragment widgets arranged in a responsive format to work for desktop as well as mobile screen size. See Work with Fragments.

Some important details for implementing the application:

  • Widgets: The dashboard page will have a filtering user interface applicable to all widgets like selecting the time frame, compartment, etc. The widgets receive these filter criteria through fragment input parameters.
  • Fragments: The code for the fragment widgets define the input parameters, which of those parameters are the required ones, and which of them are optional. The fragment listens on changes of its input parameters and refreshes data that it presents automatically upon any change by fetching the updated data according to the new filtering constraints.

    Fragment has the vbEnter event which when fired starts to run the query for Oracle Logging Analytics and stores the results into the ADP variable for the presentation layer.The listener on the fragment input parameters re-runs the same logic as vbEnter event whenever the input parameters change.

    A widget can allow interaction with the data, for example, click a chart segment, select a table row, etc. Such events are fired from the fragment using the fragment custom event to allow the parent dashboard page to respond to them in a suitable way, for example, by drilling down in a different dashboard page, by filtering existing data, etc.

    The fragment variables and input parameters are defined in the file log-records-trend-fragment.json in Oracle custom data format. Explore the fragments in Design Time for a friendly interface and update the properties. See Customize How a Fragment Variable is Displayed in the Properties Pane.

    Fragment Example for Log Records Trend Chart: log-records-trend-fragment.html

    Fragment Example for Data Loading Chain: fetchRecordsAsync.js


  • Page user interface: The page user interface presents data from ArrayDataProvider variable and typically is a chart, table, or anything suitable. What query is executed and how the resulting data are presented is the private implementation of the fragment. This can be a table component, pie chart, bar chart, summary label, etc.


Calling REST API:

This is implemented in AppModule.fetchData function using Rest.get() call. This method will implement data access layer.

More Learning Resources