Custom Logs
Custom logs are logs that contain diagnostic information from custom applications, other cloud providers, or an on-premise environment.
-
By using PutLogs to ingest custom logs directly. See the Logging Ingestion API and REST APIs for more information. Also see Using the API for an example log entry payload that can be used with PutLogs.
-
By configuring the Unified Monitoring Agent. See Installing the Agent for instructions.
Note
When managing Oracle Cloud Agent plugins, the Unified Monitoring Agent is referred to as "Custom Logs Monitoring".
Custom logs can be viewed in the Oracle Cloud Infrastructure Compute instance page, and have an associated Logs resource. They can also be viewed on the Logging Search page, Logs page, or within an associated Log Groups detail page. Custom logs are also supported in bare metal instances.
You can also view the rate at which custom logs are being accepted and ingested by using the Oracle Cloud Infrastructure Monitoring service. See Viewing the Custom Logs Acceptance Rate for more information.
The Unified Monitoring Agent can be installed on many machines, and it pulls logs from local directories, where your apps or systems emit logs. The agent can also parse your logs for you. All of this is configured in Agent Configurations. You can create an agent configuration separately, and then associate a custom log with it, or create a custom log and then later create its agent configuration.
- What hosts you want logs from.
- What specific logs you want from the hosts.
- Additional parsers.
- The custom log destination.
Creating a custom log is a two-step process, in that you create the custom log object first, and then second, create its associated agent configuration. See Creating Custom Logs for more information on creating custom logs and agent configurations, and Agent Management for more information on setting up and managing the agent.
For the agent to run correctly, ensure that your firewall settings allow the following URI endpoints:
- https://auth.<your region>.oraclecloud.com
- https://ingestion.logging.<your region>.oci.oraclecloud.com.
Creating Custom Logs
To create custom logs:
- Open the navigation menu and click Observability & Management. Under Logging, click Logs.
- Under List Scope, Compartment, choose a compartment you have permission to work in.
- Click Create custom log. The Create custom log panel is displayed.
- In Custom log name, enter a name for the custom log. Avoid entering confidential information.
- From Compartment, choose a compartment you have permission to work in.
- From Log group, select a log group to place the custom log into.
- Optionally, select a log retention value from Log Retention, and add any applicable tags in Add Tags.
- Click Create custom log. The Create agent configuration panel is displayed. You can next create a new configuration, to define the parameters for the associated log data (the default), or add it later.
- Enter a Configuration name in the corresponding field, and select a Compartment you have permissions to work in.
- In Host Groups, which allows you to define which VMs apply to
this configuration, select a Group type from the list,
whether Dynamic group or User group.
For the Dynamic group case, Dynamic Group refers to a group of instances, which you can create in the IAM feature of the Console. See About Dynamic Groups for more information. These Dynamic Groups can be selected from the Group field when setting up Dynamic Group settings.
For the User group case, select the group from the Group field. User Groups also refer to the IAM Groups feature of the Console. See Managing Groups for more information.Click + Another host group to add more groups. You can add a combination of Group Types for the agent configuration, that is, both Dynamic groups and User groups can be set up in the configuration.
Note
A maximum of five groups per configuration are allowed, and a host can be in a maximum of five different groups. - Next, in the configuration, you need to define the format of the logs (that is, what
logs do you want to watch for) in Configure log inputs.
Select an Input type form the list, whether
Windows event log or Log path.
- For Windows event log, enter an Input name and select an Event channels option from the list.
- For Log path, enter an Input name and File paths in the corresponding fields. For example, /<log_path>/<log_name>. Multiple paths can be entered.
Note
Multiple log file paths can be specified, separated by a comma (,). See https://docs.fluentd.org/input/tail#path for more information. In the configuration, you can define multiple log files separated by a comma as below:<source> @type tail tag 757261.oc_oslogs_linux path /var/log/.log,/var/log/.out,/var/log/dmesg,var/log/grubby,/var/log/messages*,var/log/secure,/var/log/auth,/var/log/acpid,/root/.bash_history pos_file /etc/unifiedmonitoringagent/pos/757261-oc_oslogs_linux.pos path_key tailed_path </source>
Example configuration:{{path C:\Program Files (x86)\<application>\<directory>*, C:\Program Files (x86)\<application>\<application_logs_directory>\<directory>* }}
Click Advanced parser options, which opens the Advanced parser options panel. This allows you to specify how to parse the log, according to the following parsers. Some of the parsers require further input and have more options, depending on the type chosen.For example for JSON, you must select a Time type value from the list, while optionally, you can specify event time and null field settings. Meanwhile for REGEXP, you specify the regular expression for matching logs, along with the time format. See Log Inputs and Parsers for more information.- AUDITD
- JSON
- TSV
- CSV
- NONEImportant
The NONE parser type is required, even if you do not want to specify a particular parser type. - SYSLOG
- APACHE2
- APACHE_ERROR
- MSGPACK
- REGEXP
- MULTILINE
- After configuring the log inputs and the parser, you can optionally specify any tag settings. Click Create custom log to save your changes, and create the custom log and its associated agent configuration.
In summary, the agent configuration defines what instances the configuration applies to (Host groups), which log files are obtained and what parser (if any) is used (Configure log inputs), and to what log object in the Oracle Cloud Infrastructure system that the records are pushed to (Select log destination). The latter is already set up since this was set during the custom log creation step.
The custom log object is now created, as well as the agent configuration, which pulls data from instances, and pushes into the custom log object.