Enabling Logs in Oracle Cloud Infrastructure Logging

Data Integration lets you use Oracle Cloud Infrastructure Logging to enable logging and retrieve service logs in the Console or from the CLI.

Public logs are available for the following types:

Execution logs for different task runs including integration, data loader, REST, SQL, and pipeline task runs
Create, edit, and delete data flow logs

Only administrative users can enable logging. Non-administrative users can only view log contents, and only after administrative users have enabled and set up logging, and given you access to logging groups and content.

Currently, Data Integration service logs are enabled at the workspace resource level using the DIS Workspace Logs log category. For example, when logging is enabled and active, integration task runs associated with data flows in a specified Data Integration workspace have the logging feature turned on.

Resource logs are grouped in logical containers called log groups in Oracle Cloud Infrastructure Logging. Administrative users must create at least one log group.

Before you start using Oracle Cloud Infrastructure Logging:

Get familiar with basic concepts and terminology used in the OCI Logging service. See Logging Overview in the OCI Logging documentation.
Create a group to manage access to log groups and log content. See Working with Groups in the OCI IAM documentation.
Add the policy to enable public logging for the Data Integration service.

Note

To restrict the enable permission to specific users, replace any-user with a specific user.
```
ALLOW any-user TO {DIS_WORKSPACE_OBJECT_CREATE} in tenancy WHERE ALL {request.service.name = 'dataintegration', request.principal.type = 'service'}
```

Add the policies to allow you to use OCI Logging with the Data Integration service.

allow group <group-name> to manage log-groups in compartment <compartment-name>
allow group <group-name> to manage log-content in compartment <compartment-name>

Make sure that you have the policies that are required to use OCI Logging. See Required Permissions for Working with Logs and Log Groups in the OCI Logging documentation.
In the OCI Logging service, create one or more log groups to organize and manage logs in logical containers. See To create a log group, Using the Console in the OCI Logging documentation.

Enabling Logs in the Console

Follow these steps in the Console to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging. Only administrative users can enable logging.

Currently, Data Integration service logs are enabled at the workspace level. To enable logging, you need to know the workspace and the name of the compartment where your workspace is located.

Open the navigation menu and click Observability & Management. Under Logging, click Logs.
Click Enable service log. The Enable Resource Log panel is displayed.
Under Select Resource, do the following:
1. Select the Resource Compartment that has the Data Integration workspace for which you want to collect service logs.
2. For Service, select Data Integration Service to enable resource logging in the service.
3. For Resource, select the name of the Data Integration workspace on which you want to enable public logs for collection.
Under Configure Log, do the following:
1. Select the Log Category to specify the type of log to create for this resource.
  
  Currently, there is only one category, DIS Workspace Logs, which contains all logs for task runs in a specified Data Integration workspace. For example, integration task runs associated with data flows.
2. In Log Name, enter a name for this log. Use the following log name guidelines.
  - The first character must start with a letter.
  - Use from 1 to 256 characters.
  - A log name is case-sensitive. Valid characters are letters (upper or lower case), numbers, hyphens, underscores, and periods.
  - Avoid entering confidential information.
Note

You can only have one log for any combination of service, resource, and log category.
If necessary, click Show Advanced Options to specify where you want to create the log.
1. Select the Compartment.
2. Select the Log Group.
  
  The OCI Logging service uses log groups to organize and manage logs in logical containers.
  
  If there are no log groups in the selection menu, click Create New Group to create one. See To create a log group, Using the Console in the OCI Logging documentation.
(Optional) Under Log Retention, you can specify how long you want to retain the log, and also apply tags for tracking.
Click Enable Log.

The Log details page is displayed. The Status shows as "Creating" and the message "Creating log" is shown under Explore Log while the log is being created. When the log is created, the Status changes to "Active." To learn more about the Log details page, see Using the Console in the OCI Logging documentation.

Enabling Logs in the CLI

Follow these steps in the CLI to use Oracle Cloud Infrastructure Logging to enable Data Integration service logging for a workspace.

Create a public log group.

Request

oci logging log-group create --compartment-id $COMPARTMENT_ID --display-name "dis-logs"

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid>",
    "operation-type": "CREATE_LOG_GROUP",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "loggroup",
        "entity-uri": "/logGroups/<log-group-ocid>",
        "identifier": "<log-group-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-23-08T18:53:29.065000+00:00",
    "time-finished": "2022-23-08T18:53:29.065000+00:00",
    "time-started": "2022-23-08T18:53:29.065000+00:00"
  }
}

Enable Data Integration service logs for the workspace.

Request

export LOG_GROUP_ID=<log-group-ocid>
oci logging log create --display-name "MyWorkspace Logs" --log-group-id $LOG_GROUP_ID --log-type SERVICE --is-enabled true --configuration file://config.json

config.json file

{
    "compartment-id" : "<compartment-ocid>",
    "source": {
        "resource" : "<workspace-ocid>",
        "service" : "dataintegration",
        "source-type" : "OCISERVICE",
        "category" : "disworkspacelogs"
    }
}

Response

{
  "opc-work-request-id": "<log-workrequest-ocid>"
}

Get the log ID of the created public log.

The log ID is the OCID that is uniquely generated for the specified resource and log category in the specified compartment.

Request

export LOG_WR_ID=<log-workrequest-ocid>
oci logging work-request get --work-request-id $LOG_WR_ID

Response

{
  "data": {
    "compartment-id": "<compartment-ocid>",
    "id": "<log-workrequest-ocid>",
    "operation-type": "CREATE_LOG",
    "percent-complete": 100.0,
    "resources": [
      {
        "action-type": "CREATED",
        "entity-type": "log",
        "entity-uri": "/logs/<log-ocid>",
        "identifier": "<log-ocid>"
      }
    ],
    "status": "SUCCEEDED",
    "time-accepted": "2022-23-08T18:58:29.065000+00:00",
    "time-finished": "2022-23-08T18:58:29.065000+00:00",
    "time-started": "2022-23-08T18:58:29.065000+00:00"
  }
}

Example of a Data Integration Public Log

The following is an example of a Data Integration service public log in JSON format.

The Data Integration fields in the output are:

Field	Description
`logLevel`	The level of the log. For example: `INFO`,`WARN/ERROR`.
`message`	The actual log message.
`objectId`	The `taskRunId` or `dataflowId`, depending on the type of operation logged. The Id is the Universal Unique Identifier (UUID) of the object for which the log is generated. For example, for a task run, it is the `TaskRunKey UUID`. For a data flow save log, it is the `dataflow UUID`.
`opcRequestId`	The `opcRequestId` of the request that came in.
`workspaceId`	The workspace OCID.

{
  "datetime": 1660885646612,
  "logContent": {
    "data": {
      "logLevel": "INFO",
      "message": "ACTION: INTEGRATION_TASK-TASK-RUN- Submit on TASK b7fba0c4-bf0d-4c29-b49f-b21673cc2791 - SUCCESS",
      "objectId": "b7fba0c4-bf0d-4c29-b49f-b21673cc2791",
      "opcRequestId": "csidb3d7aef846ab99c53689c81107f0/f11c0c8e208a49e8a1f79176bdf611a6/D4927163D0EAC4B699CD3076A17C5FD0",
      "workspaceId": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>"
    },
    "id": "2d0da51e-e02d-4ba2-b766-6255ed1b1269",
    "oracle": {
      "compartmentid": "ocid1.tenancy.oc1..<long-string-letters-12345>",
      "ingestedtime": "2022-08-19T05:07:55.326Z",
      "loggroupid": "ocid1.loggroup.oc1.phx.<long-string-letters-12345>",
      "logid": "ocid1.log.oc1.phx.<long-string-letters-12345>",
      "tenantid": "ocid1.tenancy.oc1..<long-string-letters-12345>"
    },
    "source": "ocid1.disworkspace.oc1.phx.<long-string-letters-12345>",
    "specversion": "1.0",
    "time": "2022-08-19T05:07:26.612Z",
    "type": "public.log.dis.execute"
  }
}

Tips for Searching Data Integration Logs

Use the OCI Logging Search page to search logs.

For information about how to use the Search page, see Logging Search and Viewing and Working with Search Results in the OCI Logging documentation.

Before you perform a task run (execution) log search, copy the relevant task run key in Data Integration by using the Actions menu as described in Viewing Task Runs.

Use the following tips when searching Data Integration logs:

Limit results to a narrower or specific date and time range to optimize search time.
In the results table, logs with the public.log.dis.execute type are logs from Data Integration. For better viewing, however, click X to remove the type column so that the data.message column is wider.
If a large log message is truncated in the data.message column (such as a SYSTEM.ERROR_MESSAGE), place the cursor over the message to see the full message in a box that pops open.
For a pipeline with a nested pipeline, search for the root pipeline task run key to retrieve logs for the pipeline and all its child tasks (including the nested pipeline). In such a pipeline execution log, a child task run in the nested pipeline is identified by this format string:
```
[root pipeline task run key]/[parent pipeline task run key]/[child task run key]
```
To see more detailed logs for a data loader child task run or integration task log run, copy the value of the child task run key and search for it in a separate search.

Oracle Cloud Infrastructure Documentation

Enabling Logs in Oracle Cloud Infrastructure Logging

Enabling Logs in the Console

Enabling Logs in the CLI

Example of a Data Integration Public Log

Tips for Searching Data Integration Logs