Ingest Custom Logs from OCI Logging Service Using Service Connector

If you have custom logs in OCI Logging service that you want to ingest in Oracle Logging Analytics for the purpose of analytics and log insights, then complete the steps listed here.

Note

For mission critical applications which require low latency logs ingestion and ingest time alerting, Oracle recommends that you use Oracle Management Agent which is installed by default on all OCI Instances.

If you want to collect historical logs, then use Oracle Management Agent because service connector doesn't support the collection of historical logs.

Additionally, after you install the management agent, complete the steps 1 through 5 listed below to start collecting the logs in Oracle Logging Analytics.

For more information about ingesting logs from other OCI services using service connector, see Ingest Logs from Other OCI Services Using Service Connector.

Steps to collect custom logs form OCI Logging service using service connector:

Provide permission to access and collect logs from OCI Logging service. See Allow Collection of Logs from OCI Logging Service.
Identify or create the compartment and Logging Analytics log group where the logs must be collected.

In your log record in the OCI Logging service, note the event type corresponding to the key type, for example, com.oraclecloud.logging.custom.kube_system_namespace. The value of the key will be used later to create the log source. Here's an example of a log record from OCI Logging service:

{
  "datetime": 1652152576897,
  "logContent": {
    "data": {
      "message": "2022-05-10T03:16:16.161346406+00:00 stderr F time=\"2022-05-10T03:16:16Z\" level=error msg=\"tkw network request proxied\" error=EOF",
      "tailed_path": "/var/log/pods/kube-system_proxymux-client-aaaa/proxymux-client/0.log"
    },
    "id": "id112233",
    "oracle": {
      "compartmentid": "ocid1.compartment.oc1..a....",
      "ingestedtime": "2022-05-10T03:17:25.628Z",
      "instanceid": "ocid1.instance.oc1.phx.a....",
      "loggroupid": "ocid1.loggroup.oc1.phx.a....",
      "logid": "ocid1.log.oc1.phx.a....",
      "tenantid": "ocid1.tenancy.oc1..a...."
    },
    "source": "oke-source",
    "specversion": "1.0",
    "subject": "/var/log/pods/kube-system_proxymux-client-aaaa/proxymux-client/0.log",
    "time": "2022-05-10T03:16:16.897Z",
    "type": "com.oraclecloud.logging.custom.kube_system_namespace"
  }
}

In the above example of event type, com.oraclecloud.logging.custom. is constant for custom logs. The remaining part of the value, for example, kube_system_namespace will vary depending on the type of logs.

Create a parser to parse your custom logs. Use the example log content from your logs to identify the fields. See Create a Parser.
Create a log source with the name which is same as the value of the key type. If the value you noted earlier is com.oraclecloud.logging.custom.kube_system_namespace, then the source name must be the same. While creating the log source, specify the parser that you created in the previous step. See Create a Source.
Create a new service connector or use an existing service connector which establishes a connection between OCI Logging and Oracle Logging Analytics to route your custom logs. Specify the target as Logging Analytics and add new Source Connection corresponding to your Logging Custom Logs. Note your Logging log group compartment, Logging log group and Logging custom logs details.

For detailed steps to create the service connector, see step 2 in Set Up the Service Connector to Ingest Logs.

In the Configure task section, you can set up a log filter task and specify the property to use for filtering. See Oracle Cloud Infrastructure Documentation - Manage Service Connectors.

In case of multiple custom logs, you can use the same service connector and add more log filter tasks.

After the connection is established with OCI Logging, the custom logs will start flowing into Oracle Logging Analytics. To search for you custom logs in the Log Explorer, you can query the log source name. For example,

'Log Source' = com.oraclecloud.logging.custom.kube_system_namespace | stats count as logrecords by 'Log Source'

Oracle Cloud Infrastructure Documentation