Ingest Logs from Other OCI Services Using Service Connector

You can analyze the logs to troubleshoot issues, monitor health and performance and observe the operational tasks in Oracle Cloud Infrastructure services by ingesting the logs into Oracle Cloud Logging Analytics.

Use the Service Connector to identify your Oracle Cloud Infrastructure service as the source of the logs and Oracle Cloud Logging Analytics as the destination. For information on how the Service Connector Hub works, see Service Connector Hub Overview in Oracle Cloud Infrastructure Documentation.

  • List of Oracle-defined sources for collecting logs: For the list of Oracle-defined sources to collect logs from Oracle Cloud Infrastructure services, see Oracle-Defined Sources and search for sources with title OCI...

  • Types of service logs you can collect: For the types of logs you can collect from the Oracle Cloud Infrastructure services, the parsers, example log content, fields, and JSON path, see OCI Parser Details.

  • Filter logs collected though service connector: The service connector OCID is mapped to the field Log Origin. To view the logs flowing from that service connector to Oracle Cloud Logging Analytics, filter the logs by the field Log Origin. See Filter Logs by Pinned Attributes and Fields.

Note

After the service connector is created, an entity is automatically created for processing the logs. To ensure proper log collection, the entity must not be deleted.

Permission Required for Collecting Service Logs

Based on the type of service logs that you want to ingest, you must create policies to enable Oracle Cloud Logging Analytics to get the information about the resources and create an entity for each resource.

After you create the policy, the entity that is created will be auto-associated with all the logs collected from that resource. If the policy is not created, then the logs are still ingested but the entity is not created.

Policy for each type of service logs:

  • Event Service Logs

    allow service loganalytics to {EVENTRULE_READ} in tenancy
  • Load Balancer Logs

    allow service loganalytics to {LOAD_BALANCER_READ} in tenancy
  • Object Storage Logs

    allow service loganalytics to {BUCKET_READ} in tenancy
  • Function Logs

    allow service loganalytics to read functions-family in tenancy
  • API Gateway Logs

    allow service loganalytics to read api-gateway-family in tenancy
  • Virtual Cloud Network Logs

    allow service loganalytics to {VNIC_READ} in tenancy

Set Up the Service Connector to Ingest Logs

Before you set up the service connector to ingest logs, ensure that the compartment and log group are identified for the logs that you want to ingest.

In the following example, the steps show you how to collect VCN service logs from Oracle Cloud Infrastructure Logging service:

  1. This is a suggestive step to show you how to enable logs in the Oracle Cloud Infrastructure Logging service.

    Go to Oracle Cloud Infrastructure Logging service > Go to Logs.

    Click Enable Resource Log to enable VCN service logs. The dialog box opens.

    1. Select the resource compartment.
    2. Select the service, for example, Virtual Cloud Network (subnets).
    3. Select the resource, for example, the VCN resource.
    4. Under Configure Log, select the log category, for example, Flow Logs, and the log name.
    5. Under Log Location, select the compartment and log group that Oracle Cloud Logging Analytics will refer the logs from.

    Click Enable Log.

  2. Set up the service connector by specifying the source service of the logs and the target as Oracle Cloud Logging Analytics. You can either set it up from the source service that has integrated with Oracle Cloud Infrastructure Service Connector Hub, for example, Oracle Cloud Infrastructure Logging service, or directly from Oracle Cloud Infrastructure Service Connector Hub.

    Go to Oracle Cloud Infrastructure Logging service > Go to Service Connectors > Click Create Connector.

    Alternatively, go to Oracle Cloud Infrastructure Service Connector Hub service > Click Create Service Connector.

    The Create Service Connector page opens.

    1. Enter a name for the connector and provide a description.
    2. Select the resource compartment where the connector resource must be created.
    3. Under Configure Service Connector, specify Logging as the Source service, and Logging Analytics as the Target service.
    4. Under Configure Source Connection, provide the details of the logs to collect from the service, for example, the VCN service logs.

      Select the compartment name, the log group to which the logs belong, and the name of the logs that you had configured in step 1.

    You can configure the same service connector to collect more logs. Click Another Log and repeat step 2-d.

    Optionally, you can create filters under Configure Task.

    Click Create Connector.

After the service connector is created, you can verify that the selected logs are available in Oracle Cloud Logging Analytics.

Cross-Tenancy Service Connector Hub Log Collection

Let Source_Tenant be the tenant of the source service such as Oracle Cloud Infrastructure Logging from which logs are collected. Let Destination_Tenant be the tenant in which the service connector is created. The service connector is configured with Oracle Cloud Logging Analytics as the destination for the logs that are collected from the source service. It is assumed that the service connector hub and Oracle Cloud Logging Analytics are available on the same tenant.

Set the following policies to configure the log collection from the source service when it is set up in a tenancy that is different from the tenancy of the service connector hub.

Policies To Be Added in the Source Tenant

Here is an example of policy statements which allow any user of the service connector hub tenancy to have READ access to the Logging service:

define tenancy <Destination_Tenant> as <Destination_Tenant_OCID>
admit any-user of tenancy <Destination_Tenant> to read logging-family IN TENANCY WHERE ALL {request.principal.type = 'serviceconnector'}

Ensure to set the policy for the type of service logs that must be collected from the source service. See Permission Required for Collecting Service Logs.

Policies To Be Added in the Destination Tenant

Here is an example of policy statements which allow any user to access the Logging service through the service connector hub, and the destination IAM group Destination_User_Group to have MANAGE access to the service connector hub:

define tenancy <Source_Tenant> as <Source_Tenant_OCID>
endorse any-user to read logging-family IN tenancy <Source_Tenant> WHERE ALL {request.principal.type = 'serviceconnector'}

This policy is automatically created if the service connector hub is created through OCI console. However, it must be created manually for cross-tenancy connector:

allow any-user to {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} in compartment id <Log_Group_Compartment_OCID> 
    where all 
    {request.principal.type = 'serviceconnector', 
    target.loganalytics-log-group.id = '<Log_Group_OCID>',
    request.principal.compartment.id = '<Service_Connector_Compartment_OCID>'}
allow group <Destination_User_Group> to MANAGE serviceconnectors in Tenancy

In the above policies,

  • Log_Group_Compartment_OCID: The compartment OCID of the log group in Oracle Cloud Logging Analytics where the logs must be stored.

  • Log_Group_OCID: The OCID of the log group in Oracle Cloud Logging Analytics where the logs must be stored.

  • Service_Connector_Compartment_OCID: The compartment OCID of the service connector hub.

Create a Connector Between the Source and Target Tenants

After the required policies are created for the source and destination tenants, create a service connector using CLI. See CLI Command Reference - Create.

The following example LoggingSourceDetails JSON provides source service Logging details:

{ "kind": "logging", 
"logSources": [ 
  {
     "compartmentId": "<compartment_OCID>", //compartment in the Source tenancy where Logging log group can be found
     "logGroupId": "<log_group_OCID>", // Logging log group OCID from Source tenancy in the above compartment
     "logId": "<log_OCID>"//Optional logID from Source tenancy. If not provided, all logs coming to the above log group will be picked up.
  }
 ]
}

The following example LoggingAnalyticsTargetDetails JSON provides target service Logging Analytics details:

{
  "kind": "loggingAnalytics",
  "logGroupId": "<log_group_OCID>"// Logging Analytics log group OCID from Destination tenancy
}

After the service connector is created, you can verify that the selected logs are available in Oracle Cloud Logging Analytics.