Configure the Format of the Incoming Log Events

The incoming log events must be in a specific format so that the Fluentd plug-in provided by oracle can process the log data, chunk them, and transfer them to Oracle Log Analytics.

Ensure that the following mandatory parameters are available in the Fluentd event processed by the output plug-in, for example, by configuring the record_transformer filter plug-in :

• message: The actual content of the log obtained from the input source

• entityType: The entity type with which this log data is associated

• entityName: The entity name with which this log data is associated

• sourceName: The log source name. See the list of available out-of-the-box log sources at Out-of-the-Box Log Sources.

• tag: The tag which will be used by Oracle's Fluentd plug-in to filter the log events that must be consumed by Oracle Log Analytics.

The following optional parameters can by included in the record_transformer filter plug-in:

• logEntity: The entity with which this log data is associated, typically a file name

• logMetadata: The metadata specifying the key-value pairs. Each key must be from the out-of-the-box fields available in Oracle Log Analytics or user-defined by following the steps in Create a Field. Also, to avoid the metadata pair from getting rejected during processing, ensure that the value is of the correct type.

  The fields are typically used to associate with the parse expressions.

Note that configuring the record_transformer filter plug-in is only one of the ways of including the required parameters in the incoming events. There could be other ways too.

When you use the input tail plugin @type multiline, set the parameter multiline_flush_interval to a suitable value to ensure that all the log lines are uploaded to Oracle Management Cloud in time. If the parameter is not set, then the last line of an inactive log file will be processed only when stopping the td-agent.

An example input configuration that can be used for monitoring log files from the log sources Apache HTTP Server Access Logs and Linux Syslog Logs:

<source>
	@type tail
	<parse>
		@type multiline
		multiline_flush_interval 5s
		format_firstline /([0-9A-Fa-f.:%/]+)\s+([\w\-]+)\s+([\w\-]+)\s+/
		format1 /^(?<message>.*)/
	</parse>
	path access.log
	pos_file access.log.pos
	path_key tailed_path
	tag omc.apache.access
</source>

<filter omc.apache.access>
	@type record_transformer
	enable_ruby true
	<record>
		entityType omc_host_linux
		entityName host.example.com
		sourceName "Apache HTTP Server Access Logs"
		logMetadata ${{Environment: 'test', Type: 'testMetadata'}}
		logEntity "${record['tailed_path']}"
	</record>
</filter>

<source>
	@type tail
	<parse>
		@type multiline
		multiline_flush_interval 5s
		format_firstline /^\w+\s*\d{2}\s*\d{2}:\d{2}:\d{2}\s\w+/
		format1 /^(?<message>.*)/
	</parse>
	path /var/log/messages
	pos_file var.log.messages.pos
	path_key tailed_path
	tag omc.var.log.messages
</source>

<filter omc.var.log.messages>
	@type record_transformer
	<record>
		entityType omc_host_linux
		entityName host.example.com
		sourceName "Linux Syslog Logs"
		logEntity "${record['tailed_path']}"
	</record>
</filter>

In the above example:

• The first in_tail plugin reads the logs from the tail of the log file access.log, and tags them with omc.apache.access.

• The second in_tail plugin reads the logs from the tail of the log file /var/log/messages, and tags them with omc.var.log.messages.

An example Fluentd event that adheres to the specified format:

tag: omc.apache.access
time: 1572600797
record: {
	"message": "xx.xx.xx.xx - - [14/Feb/2019:18:25:14 +0100] \"GET /administrator/ HTTP/1.1\" 200 4263 \"-\" \"Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0\" \"-\"",
	"entityType": "omc_host_linux",
	"entityName": "host.example.com",
	"sourceName": "Apache HTTP Server Access Logs",
	"logMetadata": {
		"Environment": "test",
		"Type": "testMetadata"
	},
	"logEntity": "access.log"
}