Edit Fluentd Configuration File

Edit the configuration file provided by Fluentd or td-agent and provide the information pertaining to Oracle Log Analytics and other customizations.

The output plug-in buffers the incoming events before sending them to Oracle Log Analytics. The plug-in will separate the log events into chunks by the value of the fields Tag and the sourceName. The chunks are then transferred to Oracle Log Analytics.

It is recommended that a secondary plug-in is configured which would be used by Fluentd to dump the backup data when the output plug-in continues to fail in writing the buffer chunks and exceeds the timeout threshold for retries. Also, for unrecoverable errors, Fluentd will abort the chunk immediately and move it into secondary or the backup directory. For more information, see Fluentd Documentation.

The Fluentd configuration file will be of the following format:

<match pattern>
  @type oracle_omc_loganalytics
  http_proxy                    <your_proxy>
  omc_oauth_client_id           <your_omc_oauth_client_id>
  omc_oauth_client_secret       <your_omc_oauth_client_secret>
  omc_oauth_username            <your_omc_oauth_user>
  omc_oauth_password            <your_omc_oauth_password>
  omc_oauth_scope               <your_omc_oauth_scope>
  omc_oauth_token_url           <your_omc_oauth_token_url>
  omc_oauth_upload_url          <your_omc_oauth_upload_url>
  <buffer tag, sourceName>
    @type file
    path                        <your_path_buffer_chunk_files>
    overflow_action block
  </buffer>
  <secondary>
    @type file
    path                        <your_path_backup_failed_chunks>
  </secondary>
</match>

In the above format,

http_proxy: The proxy URL, if you're using one
omc_oauth_client_id: The ID of the client application
omc_oauth_client_secret: Secret of the client application
omc_oauth_username: The user name to access Oracle Management Cloud
omc_oauth_password: The user name to access Oracle Management Cloud
omc_oauth_scope: Scope of the personal access token. To compose this URL:
1. On the Oracle Management Cloud home page, click the OMC Navigation Menu on the top-left corner and navigate to Administration > Agents.
2. On the Agents page, click the Download tab. The Agent Software Download page is displayed.
3. Select Cloud Agent from the Agent Type drop-down list. The OMC_URL is displayed. Note the URL.
4. Insert OMC_URL in the following scope URL format:
```
https://<OMC_URL>/serviceapi/%20offline_access
```
  The resulting URL is the scope.
omc_oauth_token_url: The URL to obtain the personal access token. To compose this URL:
1. Your IDCS console URL is found when you are logging in to Oracle Management Cloud. It follows the format:
```
https://<IDCS_DOMAIN_NAME>/ui/v1/adminconsole/?root=users
```
2. Obtain IDCS_DOMAIN_NAME from your IDCS console URL.
3. Insert IDCS_DOMAIN_NAME in the following token URL format:
```
https://<IDCS_DOMAIN_NAME>/oauth2/v1/token
```
  The resulting URL is the token URL.
omc_oauth_upload_url: The URL to upload the logs to Oracle Log Analytics. To compose this URL, obtain OMC_URL:
1. Obtain OMC_URL by following the same steps as for omc_oauth_scope parameter.
2. Insert OMC_URL in the following upload URL format:
```
https://<OMC_URL>/serviceapi/loganalytics/logEventsReceiver/upload
```
  The resulting URL is the upload URL.
tag: The tag that you have defined for the log event in the record_transformer plug-in, which will be used for distinguishing the log content that must be consumed by the buffer for chunking. See Configure the Format of the Incoming Log Events.
sourceName: The log source name that you have defined for the log event in the record_transformer plug-in, which will be used for distinguishing the log content that must be consumed by the buffer for chunking. See Configure the Format of the Incoming Log Events.
path in buffer section: The location where the buffer chunks are stored.
path in secondary section: The location where the backup of the failed chunks are stored. The exact location of the backup directory is determined by the parameter root_dir. See output: Backup for broken chunks in Fluentd Documentation.

For more information on the OAuth properties, see Authentication: Enable OAuth With REST API in REST API for Oracle Management Cloud.

Some of the optional properties that you can define in the configuration file:

omc_oauth_token_max_tries: Defines the maximum number of retries to get the access token. Default value is 3.
omc_oauth_token_expiration_time: The access token expiration time. The default value is 3600 sec.
omc_oauth_unauthorized_retry_time: The wait time before a retry attempt is made to obtain access token. The default value is 3.
verify_ssl: The default value is true. Disable it if you do not want to verify ssl connection.

The output plug-in uses the following configuration in the base framework of Fluentd to control the buffering and flushing behavior. It is recommended that these values are retained for better throughput:

  chunk_limit_size 1m
  flush_interval 60s
  flush_thread_interval 0.5
  flush_thread_burst_interval 0.05
  flush_thread_count 10

To override the above buffer configuration values, see Fluentd Documentation.

Here's an example configuration file with the mandatory properties where some of the parameters are defined as environment variables:

<match pattern>
  @type oracle_omc_loganalytics
  http_proxy                    "#{ENV['HTTP_PROXY']}"
  omc_oauth_client_id           "#{ENV['OMC_OAUTH_CLIENT_ID']}"
  omc_oauth_client_secret       "#{ENV['OMC_OAUTH_CLIENT_SECRET']}"
  omc_oauth_username            "#{ENV['OMC_OAUTH_USER']}"
  omc_oauth_password            "#{ENV['OMC_OAUTH_PASS']}"
  omc_oauth_scope               https://<OMC_URL>/serviceapi/%20offline_access
  omc_oauth_token_url           https://<IDCS_DOMAIN_NAME>/oauth2/v1/token
  omc_oauth_upload_url          https://<OMC_URL>/serviceapi/loganalytics/logEventsReceiver/upload
  <buffer tag, sourceName>
    @type file
    path                        <your_path_buffer_chunk_files>
    overflow_action block
  </buffer>
  <secondary>
    @type file
    path                        <your_path_backup_failed_chunks>
  </secondary>
</match>