Troubleshoot using logs

Learn to troubleshoot common issues using OCI logging and logs found in your deployment backups, and collect diagnostics from your OCI GoldenGate deployment.

Related Topics

Process and error logs

You can integrate OCI Logging with OCI GoldenGate to enable, manage, and search GoldenGate process and error logs. Integration with the Logging service is optional.

  • Error logs: Contain content from the ggserr.log file that logs processing events, messages, errors, and warnings generated by GoldenGate.
  • Process logs: Contain content from several log files for Administration Service, Distribution Service, Performance Metrics Service, Receiver Service, and Extract/Replicat Events.

Before you start using OCI Logging:

  • Get familiar with basic concepts and terminology used in the OCI Logging service. See Logging Overview in the OCI Logging documentation.
  • Create a group to manage access to log groups and log content. See Working with Groups in the OCI IAM documentation.
  • Add policies to enable public logging for OCI GoldenGate.

Note:

You manage the lifecycle of the logs including logs that are automatically created for you by OCI GoldenGate. Logs aren't deleted when the job and job runs are deleted. Learn more about Observability and management pricing.

Add policies to use OCI Logging with OCI GoldenGate

To enable service logs, you must grant your user manage access on the log group, and access to the resource. Logs and log groups use the log-group resource-type, but to search the contents of logs, you must use the log-content resource-type. Add the following policies: 

allow group <group-name> to manage log-groups in compartment <compartment-name>
allow group <group-name> to manage log-content in compartment <compartment-name>

Learn more about policies.

Enable Logging using the Oracle Cloud console

  1. On the Deployments page, select the deployment you want to enable logging for.
  2. On the Deployment details page, under resources select Logs.
  3. In the Logs table, under the Enable log column, select the Not enabled toggle to enable either Process logs or Error logs.
  4. In the Enable log panel, from the Compartment dropdown, select a compartment.
  5. For Log group, you can:
    • Select a group from the dropdown
    • Create a new group
    • Leave it blank, and a default group is automatically assigned
  6. For Log name, enter a name.
  7. For Log retention, select the number of months from the dropdown.
  8. Click Enable log.

Wait for the status to become Active. After it is active, the Diagnostic Logs feature is turned on for the process and error logs. The process and error logs are technically 'Service logs' as they come from an Oracle Cloud Infrastructure native service like OCI GoldenGate.

Enable OCI Logging using CLI

  1. Create Log group

    Request

    oci --profile <profile_name> logging log-group create --compartment-id <compartment_ocid>
      --display-name <display_name>

    Response

    { "opc-work-request-id": "<log-workrequest-ocid>" }
  2. Get Log Group ID

    Request

    oci --profile <profile_name> logging work-request get --work-request-id
      <log-workrequest-ocid>

    Response

    {  
    "data": {    
    "compartment-id": "<compartment_ocid>",    
    "id": "<log-workrequest-ocid >",    
    "operation-type": "CREATE_LOG_GROUP",    
    "percent-complete": 100.0,    
    "resources": [      
        {        
            "action-type": "CREATED",        
            "entity-type": "loggroup",        
            "entity-uri": "/logGroups/<log-group-ocid>",        
            "identifier": "<log-group-ocid>"      
        }    
    ],    "
    status": "SUCCEEDED",    
    "time-accepted": "2023-05-09T05:57:09.641000+00:00",    
    "time-finished": "2023-05-09T05:57:09.641000+00:00",    
    "time-started": "2023-05-09T05:57:09.641000+00:00"  
    }
}
  3. Create/Enable Log object

    Request

    oci --profile <profile-name> logging log create --display-name <display-name> --log-group-id
      <log-group-ocid> --log-type SERVICE/CUSTOM --is-enabled true  --configuration file://path_to_json_file

    JSON request payload

    {  
    "compartment-id": "<compartment-ocid>",     
    "source":    
        {        
            "resource": "<GoldenGate-deployment-ocid>",            
            "service": "goldengate",            
            "source-type": "OCISERVICE",            
            "category": "error_logs/process_logs"      
        }    
}

    Response

    Response 
{  
    "opc-work-request-id": "<log-workrequest-ocid>"
}
  4. Disable OCI logging

    Request

    oci --profile <profile_name> logging log update --is-enabled false --log-group-id <log-group-ocid> --log-id <log-ocid>

    Response

    {  
    "opc-work-request-id": "<log-workrequest-ocid >"
}
  5. Search OCI Logs

    Request

    oci --profile <profile_name> logging-search search-logs
      --search-query "search \"<compartment_ocid>/<log-group-ocid>/<log-ocid>\"" --time-start '2023-05-09 09:54' --time-end '2023-05-09 09:55'

    Response

    {  
    "data": {    
        "fields": null,    
        "results": [      
            {        
                "data": {          
                "datetime": 1683626095205,          
                "logContent": {            
                    "data": {              
                        "level": "INFO",              
                        "message": "Executing command '{\n    \"name\": \"report\",\n    \"reportType\": \"lag\",\n    \"thresholds\": [\n        {\n            \"type\": \"info\",\n            \"units\": \"seconds\",\n           \"value\": 0\n        }\n    ]\n}'",              
                        "processName": "adminsrvr",              
                        "resourceId": "<GoldenGate-deployment-ocid>"            
                    },            
                    "id": "20230509095455.2051683626095",            
                    "oracle": {              
                        "compartmentid": "<compartment-ocid>",              
                        "ingestedtime": "2023-05-09T09:55:05.889Z",              
                        "loggroupid": "<log-group-ocid>",              
                        "logid": "<log-ocid>",              
                        "tenantid": "<tenant-ocid>"            
                    },            
                    "source": "<GoldenGate-deployment-ocid>",            
                    "specversion": "1.0",            
                    "time": "2023-05-09T09:54:55.205Z",            
                    "type": "com.oraclecloud.goldengate.deployment.process_logs"          
                }        
            }      
        } ],    
        "summary": {      
            "field-count": null,      
            "result-count": 1    
        }  
    }
}
  6. Get Log Status

    Request

    oci --profile <profile_name> logging log get --log-group-id <log-group-ocid> --log-id
      <log-ocid>

    Response

    {  
    "data": {    
        "compartment-id": "<compartment-ocid>",    
        "configuration": {      
            "archiving": {        
                "is-enabled": false      
            },      
            "compartment-id": "<compartment-ocid>",      
            "source": {        
                "category": "process_logs",        
                "parameters": {},        
                "resource": "<GoldenGate-deployment-ocid>",        
                "service": "goldengate",        
                "source-type": "OCISERVICE"      
            }    
        },    
        "defined-tags": {      
            "Oracle-Tags": {        
                "CreatedBy": "<creater_email_id>",        
                "CreatedOn": "2023-05-09T06:24:30.279Z"      
            }    
        },    
        "display-name": "<display_name>",    
        "freeform-tags": {},    
        "id": "<log-ocid>",    
        "is-enabled": false,    "
        lifecycle-state": "INACTIVE",    
        "log-group-id": "<log-group-ocid>",    
        "log-type": "SERVICE",    
        "retention-duration": 30,    
        "tenancy-id": "<tenancy-ocid>",    
        "time-created": "2023-05-09T06:24:30.452000+00:00",    
        "time-last-modified": "2023-05-09T06:30:17.345000+00:00"  
    },  
    "etag": "cb5bb295-9954-4949-b33f-67d6da50f83f"
}
  7. Delete Log

    Request

    oci --profile <profile_name> logging log delete --log-group-id <log-group-ocid> --log-id <log-ocid>

    Response

    {  
    "opc-work-request-id": "<log-workrequest-ocid >"
}
  8. Delete Log group

    Request

    oci --profile GGSTEST logging log-group delete --log-group-id <log-group-ocid>

    Response

    {  
    "opc-work-request-id": "<log-workrequest-ocid >"
}

Enable Logging for OCI GoldenGate in the OCI Logging service

Create a log group and configure a service log in the Logginge service if you don't have one already:

  1. Open the navigation menu and click Observability & Management. Under Logging, click Log Groups.
  2. Choose a compartment you have permission to work in and click Create Log Group.

    The Create Log Group panel is displayed.

  3. Complete the following:
    • Compartment: The compartment in which you want to create the log group. This field is pre-filled based on your compartment choice.
    • Name: A name for this log group. The first character of a log group name must be a letter. For more, see Log and Log Group Names. Avoid entering confidential information.
    • Description: A friendly description.
    • (Optional) Enter tagging information.
  4. Click Create.

    The log group detail page is then displayed.

  5. Click Logs.
  6. Click Enable service log.
  7. Under Select Resource:
    1. For Resource Compartment, select the resource's compartment.
    2. For Service, select GoldenGate from the dropdown.
    3. For Resource, select a deployment from the dropdown.
  8. Under Configure log:
    1. For Log Category, select a value from the dropdown.
    2. For Log Name, enter a name.
  9. Select Show Advanced Options, and under Log Location:
    1. For Compartment, select the compartment in which to save the log.
    2. For Log Group, select a log group from the dropdown or create new group.
  10. For Log Retention, select the number of months for which to retain the logs.
  11. Click Enable log.

Deployment backup logs

You can manually backup an OCI GoldenGate deployment to Oracle Object Storage, and then download the backup locally. You can then locate the deployment logs to troubleshoot health issues.

To learn how to create a manual back up, see managing deployment backups.

The manual deployment backup contains the full GoldenGate deployment directory structure and files, including log and trail files. The directories and files that are helpful for troubleshooting include:

  • /etc: configuration
    • /etc/ogg: parameter files
  • /var: log files, checkpoint, trail files, and so on
    • /var/checkpt: checkpoint
    • /var/data: trail files
    • /var/report: report files
    • /var/log: log files
      • ER-events.log
      • ggserr.log
      • restapi.log
      • adminsrvr.log
      • pmsrvr.log
      • distsrvr-stdout.log
      • recsrvr-stdout.log
      • distsrvr.log
      • recvsrvr.log
      • extract.log
      • replicat.log

Collect diagnostics

Collect diagnostics to analyze or share information about your OCI GoldenGate deployment. The information collected can be shared with My Oracle Support if you encounter any issues.

Before you collect diagnostics, ensure that you create an OCI Object Storage bucket.
To collect deployment diagnostics:
  1. On the Deployments page, select the deployment for which to collect diagnostics.
  2. On the deployment's Details page, from the More actions menu, select Collect diagnostics.
  3. In the Collect diagnostics panel, complete the following fields, and then click Collect diagnostics:
    1. From the Bucket dropdown, select the bucket in which to save the diagnostics file. If you want to select a bucket in a different compartment, click Change Compartment.
    2. For Diagnostics name prefix, enter a short name or a few characters to prefix the diagnostics file name.
    3. (Optional) Select the Start date from which to collect system logs.
    4. (Optional) Select the End date to which to collect system logs.

      Note:

      Diagnostics collected for the OCI GoldenGate deployment contains GoldenGate logs for the entire lifespan of the deployment and are independent of the start and end dates selected.
After you click Collect diagnostics, a new field named Diagnostics displays under the GoldenGate section of the Deployment information card. It can take several minutes for the diagnostics zip file to become available to download. When it is available, a Download link appears.