Ingest Logs from OCI Object Storage Buckets

3 Ingest Logs from OCI Object Storage Buckets

You can ingest the log data from Oracle Cloud Infrastructure (OCI) Object Storage using REST API. The data obtained can be OCI audit logs, OCI flow logs, or other OCI logs stored in buckets. After the buckets are registered with Oracle Log Analytics, they are periodically polled for new logs.

To access OCI, provide the OCI authentication credentials that are different from the credentials you use for accessing Oracle Management Cloud. Therefore, register the OCI authentication credentials in the credential store of Oracle Management Cloud to establish connection.

Prerequisites:

Enable Log Collection: Access the Oracle Management Cloud console and assign license editions. Ensure that the Log Collection toggle button is ENABLED.

See Access Oracle Management Cloud and Enable License Editions in Getting Started with Oracle Management Cloud.
Object Name Prefix: The object names must have at least one prefix (directory) associated with them to process them successfully, for example, access-logs/object1 and sample/2020-10-01T01:10Z.log.gz.
Identify Buckets: Ensure that you've identified and noted the buckets in OCI Object Storage where the log data is collected. You must register these buckets with Oracle Log Analytics so that buckets are polled for log data.
For other OCI Logs (not OCI audit logs and OCI flow logs): To facilitate Oracle Log Analytics in polling for new logs and collect them periodically, ensure that the logs are stored in the OCI Object Storage buckets in the name incremented order. When the logs are stored in the buckets in the name incremented order by using the API supplied by OCI, the log name will carry the time stamp. This enables identification of new logs based on their time stamp, for example, us-pheonix-1/ad1/2019-10-04T12:50Z.log.gz.
OCI Account Information: Note the following information from your OCI account before you perform steps for registering the OCI buckets:
- OCI Region
- OCI Namespace

To register the OCI Object Storage buckets for log collection:

Create Credentials in Oracle Management Cloud credential store. See Create Credential for OCI Authentication.
After ingesting the logs, to start viewing your OCI audit and flow log data in Oracle Log Analytics log explorer, you can use out-of-the-box log sources OCI Audit Logs and OCI VCN Flow Logs. However, to monitor other OCI logs, create a new log source and parser.

See Create a Parser and Configure New Log Sources.
Configure Oracle Log Analytics to collect logs from the list of OCI Object Storage buckets:
1. Create a configuration file in the json format and store it on the local machine, for example, bucket_config.json.
```
{
  "logType": "<log_type>",
  "bucketsInfo": [
    {
      "credential": "<OCI_credential>",
      "namespace": "<OCI_namespace>",
      "region": "<OCI_region>",
      "pollSince": "<poll_time_range>",
      "logSourceName": "<log_source_name>",
      "buckets": [
        {"name":"<bucket1>"},
        {"name":"<bucket2>"}
      ]
    }
  ]
}
```
  In the above format,
  - logType: Specify AUDIT for OCI audit logs, FLOW for OCI flow logs, or OCI_LOGS_GENERIC for other OCI logs to ingest.
  - credential: The name given in the credential store to the OCI credentials that you created in step 1
  - namespace: OCI namespace collected from your OCI account
  - region: OCI region collected from your OCI account
  - pollSince: The time range from when the polling for logs must be performed. Specify one of the following:
    - BEGINNING to collect the logs from the time they began to store in the buckets
    - Absolute time from when the logs must be collected in the standard Oracle Log Analytics format. For example, 2019-12-17T00:00:00.000Z where Z is UTC time zone.
    - CURRENT_TIME to collect the logs from the time the buckets are registered in Oracle Log Analytics which is the default setting
  - logSourceName: In case of OCI audit logs and OCI flow logs, this is optional. In case of other OCI logs, specify the name of the log source that must be used in Oracle Log Analytics. To effectively use the scope of this parameter for different buckets, see the example json below.
  - buckets: The OCI Object Storage bucket names from which you want to collect the logs
  In the following example json file created in the above format, the log source name OCI Audit Logs which is global is applicable for all the buckets, but the log source name LinuxSyslogSource is local and is applicable for a single bucket:
```
{
  "logType": "AUDIT",
  "bucketsInfo": [
    {
      "credential": "John_OCI_credential",
      "namespace": "ad3n3pqrs6oc",
      "region": "us-phoenix-1",
      "pollSince": "CURRENT_TIME",
      "logSourceName": "OCI Audit Logs",
      "buckets":[
         {"name":"bucket1_name",
         "logSourceName":"LinuxSyslogSource"},
         {"name":"bucket2_name"}
      ]
    }
  ]
}
```
2. To register the buckets with Oracle Log Analytics, run the cURL command in the following format:
```
curl -X POST -k -u '<username>:<password>' -H 'X-USER-IDENTITY-DOMAIN-NAME:<identity_domain_name>' "https://<OMC_URL>/serviceapi/logan.uploads/registerOSSConfig" -H 'Content-Type:application/json' -d "@<bucket_config_json_file>"
```
  In the above format:
  - username: Your user name to access the Oracle Management Cloud account. Depending on the type of your cloud account, the username will be in one of the following formats:
    - <username> for Oracle Identity Cloud Service (IDCS) based account
    - <tenant_name>.<username> for Traditional Cloud Account
    For information on the types of cloud accounts, see About Oracle Cloud Accounts in Getting Started with Oracle Cloud.
  - password: The password to access the Oracle Management Cloud account
  - OMC_URL: Obtain OMC URL from Agents page.
    1. On the Oracle Management Cloud home page, click the OMC Navigation Menu on the top-left corner and navigate to Administration > Agents.
    2. On the Agents page, click the Download tab. The Agent Software Download page is displayed.
    3. Select Cloud Agent from the Agent Type drop-down list. The OMC_URL is displayed. Note the URL.
  - identity_domain_name: Depending on the type of your cloud account, the identity domain name will be one of the following:
    - IDCS Identity Domain: For IDCS based cloud account, typically of the format idcs-j29b928a146e4bdd7fef12a6e6a9excm. Collect this from your cloud account details page.
    - Tenant Name: For Traditional Cloud Account, typically of the format acme.
      Follow the same steps as those to obtain OMC_URL. TENANT_NAME is displayed above OMC_URL.
    For information on the types of cloud accounts, see About Oracle Cloud Accounts in Getting Started with Oracle Cloud.
  - bucket_config_json_file: The OCI properties file that you created in step a.
  An example cURL command to register the buckets with Oracle Log Analytics in case of using a Traditional Cloud Account:
```
curl -X POST -k -u 'acme.JohnDoe:john_password' -H 'X-USER-IDENTITY-DOMAIN-NAME:acme' "https://acme.example.com:4443/serviceapi/logan.uploads/registerOSSConfig" -H 'Content-Type:application/json' -d "@bucket_config.json"
```
  An example cURL command to register the buckets with Oracle Log Analytics in case of using an IDCS Cloud Account:
```
curl -X POST -k -u 'JohnDoe:john_password' -H 'X-USER-IDENTITY-DOMAIN-NAME:idcs-j29b928a146e4bdd7fef12a6e6a9excm' "https://omc-fb68f2dffe9f4a27bda5c45778f62f41.example.com/serviceapi/logan.uploads/registerOSSConfig" -H 'Content-Type:application/json' -d "@bucket_config.json"
```

After registering the buckets information, wait for the log collection to begin. Adjust the time range in your log explorer to view the data based on their time stamp. The oldest logs are collected first.

In case of errors with select few buckets, the registering action is cancelled on all the buckets listed in the configuration file.

View the Bucket Configuration

After configuring the collection of logs from OCI Object Storage buckets, you can view the configuration at any point later.

Run the cURL command in the following format:

curl -X GET -k -u '<username>:<password>' "https://<OMC_URL>/serviceapi/logan.uploads/getOSSConfig?logType=<log_type>"

In the above format:

logType: Specify AUDIT for OCI audit logs, FLOW for OCI flow logs, or GENERAL for any other OCI logs.
username: Your user name to access the Oracle Management Cloud account. Depending on the type of your cloud account, the username will be in one of the following formats:
- <username> for Oracle Identity Cloud Service (IDCS) based account
- <tenant_name>.<username> for Traditional Cloud Account.
  Follow the same steps as those to obtain OMC_URL. TENANT_NAME is displayed above OMC_URL.
password: The password to access the Oracle Management Cloud account
OMC_URL: Obtain OMC URL from Agents page.
1. On the Oracle Management Cloud home page, click the OMC Navigation Menu on the top-left corner and navigate to Administration > Agents.
2. On the Agents page, click the Download tab. The Agent Software Download page is displayed.
3. Select Cloud Agent from the Agent Type drop-down list. The OMC_URL is displayed. Note the URL.

An example output of the command:

[ {
    "bucketId": "20d11212-59de-34d5-84aa-76d04b5b7166",
    "ociCredential" : "John_OCI_credential",
    "ociNamespace" : "ad3n3pqrs6oc",
    "ociRegion" : "us-phoenix-1",
    "ociBucket" : "odu-auditlog-pull",
    "logType" : "AUDIT",  
    "createdOn" : "2019-11-08T14:50:12.058Z"
}]

Unregister the Buckets for Log Collection

At any point after the OCI Object Storage buckets are registered for log collection, you can unregister them. After unregistering, the log collection from the specified buckets is stopped. However, previously collected log data from those buckets will continue to be available in Oracle Log Analytics.

Run the cURL command in the following format:

curl -X DELETE -k -u '<username>:<password>' "https://<OMC_URL>/serviceapi/logan.uploads/unregisterOSSConfig" -d '{"bucketIds":["<bucket_IDs>"]}' -H 'Content-Type:application/json'

In the above format:

username: Your user name to access the Oracle Management Cloud account. Depending on the type of your cloud account, the username will be in one of the following formats:
- <username> for Oracle Identity Cloud Service (IDCS) based account
- <tenant_name>.<username> for Traditional Cloud Account.
  Follow the same steps as those to obtain OMC_URL. TENANT_NAME is displayed above OMC_URL.
password: The password to access the Oracle Management Cloud account
OMC_URL: Obtain OMC URL from Agents page.
1. On the Oracle Management Cloud home page, click the OMC Navigation Menu on the top-left corner and navigate to Administration > Agents.
2. On the Agents page, click the Download tab. The Agent Software Download page is displayed.
3. Select Cloud Agent from the Agent Type drop-down list. The OMC_URL is displayed. Note the URL.
bucketIds: The IDs of the buckets that you want to unregister. You can obtain this by viewing the bucket configuration using REST API. See View the Bucket Configuration.

Note that if a bucket has undergone the cycle of register > unregister > register again, then all the logs from that bucket is collected again.