Getting Started with Data Sources

Define data source queries, create an Insight detector with rules that use those queries, and attach the detector to an OCI target.

Note

If you don’t see Data Sources on the Cloud Guard console menu, contact your Oracle account manager or Oracle Support.

What Are Data Sources?

A data source is a Cloud Guard resource that uses OCI Logging service queries to extract specified information from various log files. Cloud Guard data sources are composed of a query that runs regularly on an OCI Logging service log object that might contain any kind of log source. Log sources include Audit, VCN Flow logs, WAF logs, Custom logs, and other types of logs that are supported by OCI Logging service. Results of the query execution are correlated, based on the conditions (keys and additional entities) that you specify in the query. Correlated query results then appear as Cloud Guard problems.

Data Source Concepts

Become familiar with the following concepts.

  • A data source defines a source information upon which Cloud Guard can drive detections.
  • A query defines a Logging service search that extracts records from various log files. Queries can be created and saved in the Logging service, then imported into Cloud Guard.
  • A key is an entity that can be set as a certain log result field upon which correlation is exercised within Cloud Guard.

    Keys are defined as: cgkey01, cgkey02, cgkey03, ..., cgkey15.

    For example, a key can be a username (username as cgkey01) or hostname (hostname as cgkey02) field. When a query executes, signals based on log results are correlated against cgkey01 (username) and cgkey02 (hostname) and a unique Cloud Guard problem is created for each key or key-set value.

  • An additional entity is similar to a key, but no correlation is run against additional entities. For example, an additional entity can be a compartment name, an IP address, or any other field that's part of log record, which can provide context to the problem that created from the query.

    Additional entities are defined as: cg01, cg02, cg03, ..., cg15.

    Note

    The total number of keys and additional entities in a query must be 15 or less.
    • The minimum number of keys is one.
    • Additional entities are optional.
  • A region is where the query is created.

    You can select multiple regions while creating a data source query. The created query covers all selected regions.

  • A trigger definition specifies the frequency with which a data source query runs, and how soon it starts to run after it's created.

A Data Source Query Example

Data sources primarily include Logging service queries and other metadata, such as regions to be replicated. The Logging query typically requires some changes to allow problems to be created with the correct correlation and data context.

Review this data source query example to better understand how a data source query builds on a query imported from the Logging service.

The search part of the query determines which logging records are returned. The select clause determines what parts of the logging records that you see in the Cloud Guard problem. Cloud Guard needs the as phrase for items in the select clause in order to present the information returned on the Problem Details page for the resulting problem. For more information on the query language, see Logging Query Language Specification.

  1. When creating a data source query in the Create query dialog box, you can import a query that’s been saved in the Logging service. An imported query looks something like this in the Query text box:
    search
          "ocid1.tenancy.oc1..7teeb4dpem5bq6yffgxub2jeyjaaaaaaaakntoxgj2vt6wfsmwo75cnmshmf/_Audit" |
          (type='com.oraclecloud.objectstorage.createpar') | (data.identity.principalName='joeuser') |
          sort by datetime desc | select data.identity.principalName, data.additionalDetails.bucketName,
          data.additionalDetails.bucketId
  2. To refine this query for use in Cloud Guard, add the as… phrases to the select clause as shown in bold:
    search
          "ocid1.tenancy.oc1..7teeb4dpem5bq6yffgxub2jeyjaaaaaaaakntoxgj2vt6wfsmwo75cnmshmf/_Audit" |
          (type='com.oraclecloud.objectstorage.createpar') | (data.identity.principalName='joeuser') |
          sort by datetime desc | select data.identity.principalName as cgkey01,
          data.additionalDetails.bucketName as cg01, data.additionalDetails.bucketId as
            cg02

On the Create Query page for the example above, you would set:

  • Keys = 1 (the default), because your query uses only one key (cgkey01)
  • Additional entities = 2, because your query uses two additional entities (cg01 and cg02)

Prerequisite Tasks for Data Sources

Complete these tasks before you proceed.

  1. Enable and Configure OCI Logging Service

    Set up the OCI Logging service in your OCI tenancy and configure queries that identify issues that you want to extract from the log data. See Searching Logs in the Logging service documentation.

    The instructions in this document only provide information on tasks that you perform in Cloud Guard.

  2. Enable Cloud Guard

    See Enabling Cloud Guard.

  3. Implement Policy Statements

    Cloud Guard requires these permissions to enable data connections between Cloud Guard and the OCI Logging service:

    • allow service logging to {LOG_DEFINITION_READ, LOG_DEFINITION_WRITE, LOG_WRITE, LOG_NAMESPACE_READ, LOG_CONTENT_READ, AUDIT_EVENT_READ,LOG_CONTENT_PUSH} in tenancy
    • allow service logging to {INTERNAL_AUDIT_EVENT_READ} in tenancy
  4. Plan Your Data Source Queries

    The instructions provided in this document assume that you have worked out the details of the key Logging services searches that provide the information you want to extract from various logs.

    Cloud Guard provides the capability for you to create data source queries from scratch. However, it’s expected that most users create and refine their queries in the Logging service. After you import a saved Logging service query into Cloud Guard, you can assign the keys and additional entities that Cloud Guard requires, if those were not in the imported query.