Agent Management

To ingest events from your applications into your custom log, you can install the Oracle fluentd-based agent. This agent allows you to control exactly which logs you want to collect, how to parse them, and more.

Note

The Unified Monitoring Agent is a fully managed agent, and custom client configuration is not officially supported. For example, gathering logs from remote sources is not recommended, since doing so can have serious security implications (because the log source cannot be verified).

Oracle Cloud Infrastructure Logging provides an easy mechanism (Agent Configurations) to enable and manage the agent for a set of supported operating systems. Agent Configurations give you a central experience to easily configure what custom logs you want to ingest across your fleet of hosts. The following are the supported operating systems for agent configurations:

  • Oracle Linux 7, Oracle Linux 8
  • CentOS 7, CentOS 8
  • Windows Server 2012 R2, Windows Server 2016, Windows Server 2019
  • Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04
Note

For Linux, only register Linux-specific input types, such as "Log Path", for a dynamic group that includes only a Linux instance. For Windows, only register Windows-specific input types, such as "Windows event log", for a dynamic group that includes only a Windows instance. Otherwise, the Unified Monitoring Agent malfunctions if you register a Windows input type for a Linux instance, and vice versa.

Installing the Agent

New Oracle Cloud Infrastructure Instances

For supported operating systems, you can enable the agent directly during creation time. The Custom Logs Monitoring plugin must be enabled, and all plugins must be running. See Managing Plugins with Oracle Cloud Agent for more information.

Existing Oracle Cloud Infrastructure Instances

For existing instances with supported operating systems, the Custom Logs Monitoring plugin must be enabled, and all plugins must be running. See Managing Plugins with Oracle Cloud Agent for more information.

If you already have the monitoring plugin enabled, then your instance will be automatically patched to install the agent by September 18, 2020. Otherwise, you can follow the manual installation instructions:

Linux:

  1. Connect to the instance.
  2. You can obtain the agent by using this script, and then run the script to download the agent. For example:

    ./downloadAgent.sh centos7 
    ./downloadAgent.sh centos8
    ./downloadAgent.sh oel7
    ./downloadAgent.sh oel8
    ./downloadAgent.sh windows
    ./downloadAgent.sh ubuntu16
    ./downloadAgent.sh ubuntu18
    ./downloadAgent.sh ubuntu20

    Otherwise, you can download the agent for the following individual operating systems:

    For the FIPS-enabled agent:

    Note

    Performance is impacted when using the FIPS-enabled agent.
  3. Run the following command to install the RPM:
    yum install -y <rpm-name>

    For Ubuntu:

    apt install -y <deb-package-name>

Windows:

  1. Connect to the instance.
  2. Download the agent from: https://objectstorage.us-phoenix-1.oraclecloud.com/n/axmjwnk4dzjv/b/unified-monitoring-agent-windows-repo/o/unified-monitoring-agent-0.0.7.msi
  3. Open an elevated command prompt (running as Administrator), and run the MSI command (installation can take up to five minutes to complete):
    C:\path\to\file\unified-monitoring-agent-0.0.2.msi
  4. For a more advanced version of the preceding command (to debug MSI installation issues), run:
    msiexec /i "C:\path\to\file\unified-monitoring-agent-0.0.2.msi" /l*v "C:\unified-monitoring-agent_msi.log"

Instances Created from Custom Images and Non-Oracle Cloud Infrastructure Instances

  1. Install the agent according to the same steps in Existing Oracle Cloud Infrastructure Instances.
  2. Configure User API keys for the instance you are running on. To generate the user API key, follow the instructions described in How to Generate an API Signing Key.
    • (Linux) Step 2a. Place the ".oci" directory and its contents under /etc/unified-monitoring-agent.
    • (Windows) Step 2a. For Windows, there are a few steps that differ, so ensure to follow the appropriate steps. Create the ".oci" folder and its contents in the directory C:\oracle_unified_agent.
  3. Follow the instructions described in 2. Create a Profile in the Oracle Cloud Infrastructure CLI Configuration File, to create the configuration file with the modifications in the next step.
  4. After following the steps in 2. Create a Profile in the Oracle Cloud Infrastructure CLI Configuration File, ensure to name the profile (<profile-name>) for this section as "UNIFIED_MONITORING_AGENT".
    The following is an example of what the configuration looks like for the Unified Monitoring Agent to use for authentication with the service:
    [UNIFIED_MONITORING_AGENT]
    user=ocid1.user.region..aaa...
    fingerprint=<cert fingerprint>
    key_file=/path/to/ocifolder/.oci/private.pem
    tenancy=ocid1.tenancy.region..aaa...
    region=<instances region>
    pass_phrase="pashphrase1234"

Verify Agent Installation

Windows:

  1. Connect to the instance.
  2. Open Services.msc (Start menu and type services.msc). Scroll until you see the "Oracle Unified Monitoring Agent" and that the agent is in a "Running" state.
  3. In the Task Scheduler under Task Scheduler Library, verify that the UnifiedAgentConfigUpdater exists, and has (or will) run successfully. After the initial install, it can take up to 20 minutes for the first run. If preferred, this can be run manually.
  4. After the UnifiedAgentConfigUpdater task has run, verify that a "unified-monitoring-agent.conf" file in exists in C:\oracle_unified_agent.
  5. After a few minutes, supervisor (unified-monitoring-agent-supervisor-0.log) logs and worker (unified-monitoring-agent-0.log) logs appear in the C:\oracle_unified_agent directory.
  6. The preceding logs contain the Fluentd parser and plugin output.

Oracle Linux 7, Oracle Linux 8, CentOS 7, CentOS 8, Ubuntu 16, Ubuntu 18, and Ubuntu 20:

  1. Connect to the instance.
  2. Check that the agent is running by running the following command:
    systemctl status unified-monitoring-agent
  3. The status looks like the following:
    Loaded: loaded (/usr/lib/systemd/system/unified-monitoring-agent.service; enabled; vendor preset: disabled)
    Active: active (running) since Thu 2020-09-10 18:11:45 GMT; 2h 14min ago
    Docs: https://docs.cloud.oracle.com/

Managing Agent Configurations

To use agent configurations, you must be running an Oracle Cloud Infrastructure instance with the supported operating system (see Agent Management). Agent configurations give you a central experience to easily configure what custom logs you want to ingest across your fleet of hosts. A configuration allows you to select:
  • Which hosts you want to collect logs from.
  • Exactly which logs you want to ingest from those hosts.
  • A log group/log destination.
Configurations are managed through the Console and Logging API. In addition, since you can choose to create an agent configuration later after creating a custom log, you can use the Agent Configurations page to set up the agent configuration and point it to your custom log.

The Agent Configurations page is organized in terms of the following:

  • Name
  • Config OCID
  • Status
  • Created
To create a new agent configuration
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click Create Agent Config. The Agent Configurations panel is displayed.
  4. In Name and compartment, enter a Configuration Name in the corresponding field, and select a Compartment you have permissions to work in. Avoid entering confidential information.
  5. In Choose Host Groups, select a Group Type from the list, whether Dynamic Group or User Group. Click Add Host Group to add more groups.
  6. In Configure Log Inputs, select an Input Type form the list, whether Windows Event Log or Log Directory.
    • For Windows Event Log, enter an Input Name and select an Event Channels option from the list.
    • For Log Directory, enter an Input Name and a Path in the corresponding fields.
  7. In Select log destination, the User Group or Dynamic Group in the configuration that you select in Compartment needs to have permission to work in the compartment. Select the Log Group, and the Log Name from the corresponding drop-down lists. The Log Name can only point to a custom log and the custom log must exist in the chosen log group for the configuration to work.
  8. Optionally, after clicking Show Additional Options, specify any preferred tag settings.
  9. Click Create. The agent configuration is created and appears in the Agent Configurations page.
To view an agent configuration
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click the linked agent configuration name under Name in the table. The agent configuration detail page is displayed. This page displays the following on the Log Information tab:
    • OCID
    • Compartment
    • Created date and time in UTC format
    • Status (Creating, Active, Updating, Inactive, Deleting, Deleted)
    • The Tags tab shows associated tags for this log.
    • Under Log Details the following is displayed: the compartment, the linked log group and log name.

      In the Configuration resource, the Host Group and Log Input configuration settings are listed in corresponding tables. Under Host Group you can view the Group Type, Group Name, and the OCID. Click the linked Group Type (whether a User Group or Dynamic Group), which opens the IAM Groups or Dynamic Groups section of the Console, respectively. See Managing Groups and About Dynamic Groups for more information.

      Under Log Input you can view the Input Type, Input Name, File Paths, Parser, and Parser Parameters (if applicable for the chosen parser).

      In the Explore Log resource, log data is displayed in a similar manner as the Log Data on the Search page. You can apply some simple filters, such as sorting by newest or oldest from the Sort field, or filtering by time from the corresponding Filter by Time field.

      Clicking Explore with Log Search allows you to view this log on the Search page directly. After clicking this link, the Search page opens with the Select Logs to Search field populated with the log in the filter settings. At this point, you can perform more analysis and investigation related to this log directly on the Search page. For more information, see Searching Logs.

To edit an agent configuration
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click the linked agent configuration name under Name in the table. The agent configuration detail page is displayed.
  4. Click Edit. The Agent Configurations panel is displayed.

    From the main Agent Configurations page, for the agent configuration you want to edit, you can also click the the Actions icon (three dots), and then click Edit to access the Agent Configurations panel.

  5. Make your changes and click Update.
To enable or disable an existing agent configuration
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click the linked agent configuration name under Name in the table. The agent configuration detail page is displayed.
  4. Click Disable/Enable. A confirmation dialog is displayed regarding the disabling or enabling of the agent configuration.
  5. Confirm by clicking Disable/Enable. The agent configuration detail page changes its status and displays Inactive (for a disabled configuration), or Active (for an enabled configuration) in the status field, both on the agent configuration detail page and the Agent Configurations page.

    From the main Agent Configurations page, for the agent configuration you want to disable/enable, you can also click the the Actions icon (three dots), and then click Disable/Enable.

To delete an agent configuration
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click the linked agent configuration name under Name in the table. The agent configuration detail page is displayed.
  4. Click Delete. A confirmation dialog is displayed regarding the delete operation.
  5. Confirm by clicking Delete. The agent configuration is removed from the Agent Configurations page.

    From the main Agent Configurations page, for the agent configuration you want to delete, you can also click the the Actions icon (three dots), and then click Delete.

To move an agent configuration to a different compartment
  1. Open the navigation menu and click Observability & Management. Under Logging, click Agent Configurations. The Agent Configurations page is displayed.
  2. Under List Scope, Compartment, choose a compartment you have permission to work in.
  3. Click the linked agent configuration name under Name in the table. The agent configuration detail page is displayed.
  4. Click Move Resource. The Move Resource to a Different Compartment dialog is displayed.
  5. Choose the new compartment and then click Move Resource.

    From the main Agent Configurations page, for the agent configuration you want to move to a new compartment, you can also click the the Actions icon (three dots), and then click Move Resource.

Selecting Target Hosts with Dynamic Groups

To set up a configuration for multiple hosts, you can use the Dynamic Group feature from IAM. The overall process first involves creating a compartment, then placing all the instances in the compartment you want to collect logs from. Next, you can create the Dynamic Group. The Dynamic Group policy statement would then point to the compartment that contains the instances. Lastly, create a log group, custom log, and its associated agent configuration.

Set the following policy statement:
allow dynamic-group <dynamic_group_name> to use log-content in tenancy

This policy statement allows the agent configuration to push logs to the Logging service backend, which you can later see in the Logging Service's Search page.

In the Dynamic Groups configuration, set up your Dynamic Group to have a rule that includes all the agents that you want to use to send logs to the Logging service. For example, in a Rule inside the Dynamic Group it can state:

ANY {instance.id = 'ocid1.instance.<region>.<location>.<unique_ID>', 
instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}
If you remove instance.id = 'ocid1.instance.<region>.<location>.<unique_ID>' and just have:
ANY {instance.compartment.id = 'ocid1.compartment.<region>..<unique_ID>'}
this means use all the instances under this compartment to send logs. For more information on Dynamic Groups, see About Dynamic Groups.

Next, create the log group (see To create a log group). After the log group is created, you can then create the custom log and the agent configuration (see Creating Custom Logs for steps to create the custom log and agent configuration). During the agent configuration, you can use the Dynamic Group you created earlier and select it in the Choose Host Groups section of the Agent Configurations panel. This links the log configuration with the instance you want to send logs to. Once the agent configuration is active, the logs you see are sent by the instance, inside the Dynamic Group you earlier set up. You can later click Explore with Log Search in the agent configuration to view the logs through the Search page (see Searching Logs).

Log Inputs and Parsers

Agent configurations allow you to easily select which types of logs you want to ingest, and how to parse them. The following are the supported log inputs in agent configurations:

  • Windows Event Logs
  • Log Directory (Tail)

For Log Directory inputs, you can specify parsers to structure your logs. The following are the list of supported parsers:

Log Destination

You can choose the exact log group and log object where you want your log events to be indexed. All incoming log events from your hosts are ingested and indexed in your selected log object. After they are ingested, you can view and search your log events on the Search page (see Searching Logs). All existing Oracle Cloud Infrastructure Identity and Access Management policies in both the log group and compartment apply both during ingestion and search. So, only authorized users can view and ingest logs in your tenancy.