Set Up Continuous Log Collection From Your Hosts

To continuously collect log data from your entities, install the Management Agent on your host. Before that, ensure that you have completed the prerequisite tasks for using the Management Agents.

Topics:

Permission Required for Setting Up Continuous Log Collection

When you perform the prerequisites for deploying Management Agents in the step Install Management Agents, you will create the required compartment, user group for Logging Analytics users, and create IAM policies to install the Management Agents. As part of the prerequisites, ensure that the following policies are created for your user group:

ALLOW GROUP Logging-Analytics-User-Group TO MANAGE management-agents IN COMPARTMENT <compartment_name>
ALLOW GROUP Logging-Analytics-User-Group to MANAGE management-agent-install-keys IN TENANCY
ALLOW GROUP Logging-Analytics-User-Group TO READ METRICS IN COMPARTMENT <compartment_name>
ALLOW GROUP Logging-Analytics-User-Group TO READ USERS IN TENANCY

In the above example policy statements, Logging-Analytics-User-Group is an example user group.

Also, create a dynamic group for the Management Agents if it already doesn't exist, for example Management-Agent-Dynamic-Group:

ALL {resource.type='managementagent', resource.compartment.id='<management_agent_compartment_OCID>'}

Create IAM policies for Management-Agent-Dynamic-Group to enable log collection and metrics generation:

ALLOW DYNAMIC-GROUP Management-Agent-Dynamic-Group TO USE METRICS IN TENANCY
ALLOW DYNAMIC-GROUP Management-Agent-Dynamic-Group TO {LOG_ANALYTICS_LOG_GROUP_UPLOAD_LOGS} IN TENANCY

Grant READ Access of the Logs to the Agent User on Your Host

While deploying the management agents for using Oracle Cloud Logging Analytics on UNIX-based hosts, ensure that the management agent has the correct privileges to read the log files from where data has to be collected.

You can use one of the following ways (in order of best practice) to make the log files readable to the management agent:

  • Use Access Control Lists (ACLs) to enable the cloud agent user to read the log file path and log files. An ACL provides a flexible permission mechanism for file systems. Ensure that the full path to the log files is readable through the ACL.

    To set up an ACL in a UNIX-based host:

    1. Determine whether the system that contains the log files has the acl package:

      rpm -q acl

      If the system contains the acl package, then the previous command should return:

      acl-2.2.39-8.el5

      If the system doesn’t have the acl package, then download and install the package.

    2. Grant the management agent user READ access to the required log file:

      setfacl -m u:<agentuser>:r <path to the log file/log file name>

      Grant the cloud agent user READ access to the leading path or folders by running the following command:

      setfacl -R -m u:<agentuser>:r <path to the parent folder of the log file>
  • Place the management agent and the product that generates the logs in the same user group, and make the files readable to the entire group. Restart the agent.

  • Make the log files readable to all users. For example, chmod o+r <file>.

    You may have to give executable permission to the parent folders. For example, chmod o+rx <parent folder>.

Install Management Agents

See Oracle Management Agents Documentation to complete the following tasks:

  • Perform prerequisites for deploying Management Agents

  • Install Management Agent

After you install the Management Agent, complete the following Logging Analytics specific tasks to start the log collection:

  • Map your entities to your agent: Create your entities and select the Management Agent that was installed to associate the agent with this entity. See Create an Entity to Represent Your Log-Emitting Resource. You can also edit an existing entity and add the agent.

  • Configure source-entity association. You can use the Add Data wizard to perform this task. For step-by-step help to complete the task, see OCI Logging Analytics: Set Up Continuous Log Collection (Tutorial icon Tutorial ).

Note

The management agent connects to the following endpoints for Oracle Cloud Logging Analytics operations:
  • Upload the logs and log collection warning:
    https://loganalytics.<region>.oci.oraclecloud.com/<additional_part_pertaining_to_the_operation>
  • Metrics:
    https://telemetry-ingestion.<region>.oraclecloud.com/<additional_part_pertaining_to_the_operation>

In the above endpoints, region is the identifier for your region, for example, us-ashburn-1.

View Agent Collection Warnings

Oracle Cloud Logging Analytics lets you view the warning messages generated during log collection using the management agent. This helps you to diagnose problems with the sources or entities and to take corrective action.

After the cause of a collection warning is addressed, it is cleared from the list and will not be reported.

Following are the types of collection warning messages that are displayed:

  • Connection Identifier Is Empty

  • Credential Can Not Be Accessed

  • Credential Corrupted

  • Credential Is Not Enabled

  • Credential Not Found

  • Credential Store Not Found

  • Database Connection Can Not Be Established

  • Invalid Sequence Column

  • Cannot read file

  • Agent Configured to Monitor Too Many Files

  • Missing File Permission

  • File not found

  • Cannot Open Port

  • Too Many Historic Files

  • SQL Query Execution Error

Topics:

View Agent Collection Warnings Details

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Agent Collection Warnings.

    The Agent Collection Warnings page opens. This displays the list of warnings generated while collecting logs on the agent side.

    Use the multiple filters available in the left pane like Start Date, End Date, Entity Type, Source, Warning Message, and Warning State to narrow down your search for the warning messages.

    The Start Date and End Date filters use the First Reported information of the warning message to help in filtering.

    The Source Pattern that is displayed adjacent to the warning message is the one which is associated with the problem among the multiple patterns defined for that source.

    Hover the cursor on the warning message to view more details about the warning.

  3. Optionally, you can hide a warning if you want to temporarily ignore it and address it at a later point in time. Click Actions Actions menu icon > click Hide.

    Alternatively, you can select multiple warnings and hide them using the Hide Warnings button. Use the Warning State filter in the left navigation pane to view the hidden warning messages. You can move the hidden warnings back to the active list by using the Actions Actions menu icon and clicking Unhide.

  4. Additionally, in response to a warning, if you want to remove the association between its entity and source, then click Actions Actions menu icon > click Remove Association. The management agent stops collecting logs from that source and entity after removing the association. Then, the warning gets automatically cleared.

View Agent Collection Warnings in Entity Detail or Source Detail Page

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources.

    Click the name of the <resource> whose warning information you want to view. The <resource> can be Entities or a Sources.

  3. In case of Sources, the Sources page is displayed. Click the name of source whose warnings summary you want to view. The Source Detail page is displayed.

    In case of Entities, the Entities page is displayed. Click the name of entity whose warnings summary you want to view. The Entity Detail page is displayed.

  4. Click Agent Collection Warnings in the Resources section.

    The warnings summary is displayed. If you are viewing the warnings for the source, then you can see the associated entity and the entity type in the warnings summary. If you are viewing the warnings for the entity, then you can see the associated source in the summary.

As in the case of Agent Collection Warnings page, you can hide or unhide the warnings, and remove association between the source and entity. For more information, see View Agent Collection Warnings Details.

Use the filters in the left navigation pane to narrow down your search for the warning messages.

Monitor Your Continuous Log Collection

After you complete the set up for continuous log collection, the Management Agent installed on your host emits information about the size of log data that it is uploading to Logging Analytics and errors encountered, if any.

This data is displayed for each log source with the following agent log collection metrics:

  • Agent Data Upload Size (logCollectionUploadDataSize): The size of the log data collected through the Management Agent for each log source.

  • Agent Data Upload Errors (logCollectionUploadFailureCount): The count of errors occurred for each log source during the log collection and the type of errors.

To access the Agent Data Upload Size and Agent Data Upload Errors metrics, see Monitor Logging Analytics Using Service Metrics.

To modify the filters applied on the metrics data, you can view the metrics in the metrics explorer and change the metrics dimensions:

  1. Click the Options menu on the top right corner of the agent log collection metric, and select View in Metric Explorer.

    The metric is now displayed in the Metrics Explorer. Here, you can view the chart in finer detail.

  2. Click Edit Queries and select Dimension Name and Dimension Value for the metric. For example, if you want to view the upload data size for a specific host host123, then select the metric name logCollectionUploadDataSize, dimension name as agentHostName and the dimension value as host123.

    Click Update Chart to refresh the chart visualization. The chart will now display only the upload data size for the specified host.

    Similarly, if you want to view the number of upload errors encountered of the type LogGroupPolicyError, then select the metric name logCollectionUploadFailureCount, dimension name as errorCode and the dimension value as LogGroupPolicyError.

    Click Update Chart to refresh the chart visualization. The chart will now display the count of upload errors of the specified type for the specified period.

    You can switch to the Data Table view for a tabular representation of the data points in the metrics.

Following are the dimensions available to filter the metric data:

Dimension Metrics Details
agentHostName logCollectionUploadDataSize, logCollectionUploadFailureCount

The name of the host on which Management Agent is installed

logGroup logCollectionUploadDataSize, logCollectionUploadFailureCount

The log group in which the log collection happens

logSourceType logCollectionUploadDataSize, logCollectionUploadFailureCount

The log source type, which can be

  • File
  • Syslog Listener
  • Database
  • Windows Event System
  • Oracle Diagnostic Log (ODL)
resourceId logCollectionUploadDataSize, logCollectionUploadFailureCount

The OCID of the Management Agent

errorCode logCollectionUploadFailureCount

The error reported by the Management Agent

Following are the various types of errors reported by the Management Agent in the logCollectionUploadFailureCount metric for the dimension errorCode:

Error Type Description Recommended Fix

LogGroupPolicyError

Occurs due to authorization failure during log upload. This is caused by incorrect IAM policies.

HTTP status code: 404

Check the IAM policies you created for enabling continuous log collection and verify that the required permissions are given. See Permission Required for Setting Up Continuous Log Collection.

InvalidParameter

Occurs when the Management Agent sends request with incorrect parameters.

HTTP status code: 400

Contact Oracle Support with the Error Type information.

NotAuthenticated

Occurs when the Management Agent sends request with incorrect signature.

HTTP status code: 401

RequestEntityTooLarge

Occurs when the Management Agent sends request with a payload which is larger than expected.

HTTP status code: 413

TooManyRequests

Occurs when the Management Agent sends requests which are more in number than what is defined in the endpoint configuration.

HTTP status code: 429

InternalError

Occurs when an unexpected exception crops up in the Management Agent.

HTTP status code: 500

HTTP Error Code <error code>

All other unexpected error codes returned on the log upload endpoint.

For the actions that you can perform with each metric, see Actions for Service Metrics.