Manage Storage

The log data ingested into Oracle Logging Analytics are available in the active storage for analysis and display. You can optimize your active storage use by performing any of the following storage related activities based on your need:

  • Purge Logs: You can purge the unused or old log data to reduce the size of the active storage that you are consuming. You can perform purge on-demand or create a purge policy. See Purge Log Data.

  • Archive Logs: If you are unlikely to use your old logs for analysis, then enable archiving and specify the number of days from the log's timestamp after which the log data must be automatically moved from active storage to archive storage which is available at a lesser cost. You can also recall the archived log data for active use. See Archive Log Data.

  • Recall Archived Logs: After the log data is archived, you can recall the selected log data for active use. The logs are selected for recall by specifying the time range in which the timestamps of the logs are present. You can release the recalled logs back to the archive pool after active use. Note that the recalled data will count towards your active storage usage until you release it. See Recall Archived Logs.

  • Release Recalled Logs: Use this option for releasing the recalled logs back into the archive storage to optimize your storage cost. See step 7 in Recall Archived Logs.

  • View Storage Activity Report: Use this single-pane window to keep track of all your storage management activities and to perform more management tasks. See View Storage Activity Report.

Note

Your archive policy and recall activity may not complete if the time lines overlap with the purge policy. Make sure to review your purge policy and archival setting to avoid losing log data that must be archived.

Purge Log Data

Oracle Logging Analytics lets you purge log events that were loaded by agent or by an on-demand upload, to reduce the index size of the log data.

Purging enables you to bring down your usage to reduce overage charges. Oracle Logging Analytics can purge log data automatically per a set schedule or manually based on your need. Before you purge log data, create IAM policies to set up permissions for the task. See Allow Users to Purge Log Data.

There are multiple ways to purge log data.

  • By purging on-demand: All log data from the specified compartment created prior to the selected time range gets purged.
  • By creating a purge policy: The old log data can be purged by specifying a schedule for purging and the query to filter the data to purge. If you want to automate the purge activity, then you can create a purge policy by specifying the purge schedule, selecting the log data to purge, and enabling the policy.
  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

    The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  2. In the Storage page, you can purge log data in one of the following methods:
    • Perform on-demand Purge:

      Click Purge Logs. The Purge Logs dialog box is displayed.

      • Select the compartment in which the logs must be purged.

      • Specify if the subcompartments also must be included in the Purge.

      • Select the date and time prior to which the log data that must be purged was collected.

        Purge action is performed on the log data from all the buckets in the selected compartment which were collected prior to the specified time period. For example, if you specify the date and time as November 2, 2020 12:00:00 and compartment Analyze, then the log data with the time stamp older than November 2, 2020 12:00:00 stored in the compartment Analyze is deleted.

      • In the Query field, enter the query to select a specific set of log data. For example, to select the logs from the entities of the type Linux Host, specify the query 'Entity Type'='Host (Linux)'.

      • Click Estimate Reclaimed Storage to determine the size of the storage that can be reclaimed based on the selection you made in the previous fields.

      • Click Purge.

    • Create a purge policy to purge logs based on a query or age:

      Under Purge Policies, click Create. The Create Purge Policy dialog box opens.

      • Enter a name for the new purge policy.

      • Under Purge Logs Older than, select the time period from when the log data must be purged.

      • Under Schedule Interval, select the periodicity, and time of the purge action.

      • In the Query field, enter the query to select a specific set of log data. For example, to select the logs from the source Apache HTTP Server Access Logs, specify the query 'Log Source'='Apache HTTP Server Access Logs'.

      • Click Create.

      The purge policy is created.

    To delete a policy, click Actions icon Actions icon next to the policy name > click Delete.

    To view the purge activities performed, in the Storage page > under Resources, click Activity Report. The Activity Report page is displayed which summarizes all the storage activities. Use the Status and Time filters to view the preferred purge activities.

Allow Users to Purge Log Data

To purge log data, first set up right permissions by creating the following IAM policies:

  1. Create a dynamic group to allow purges for the compartments you want to allow purges in:

    ALL {resource.type='loganalyticsscheduledtask', resource.compartment.id='<compartment ocid>'}

    Alternatively, to allow purges on all compartments:

    ALL {resource.type='loganalyticsscheduledtask'}
  2. Create policies to allow the dynamic group to perform purge operation:

    allow dynamic-group <group_name> to read compartments in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_PURGE} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_STORAGE_WORK_REQUEST_CREATE} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_LOG_GROUP_DELETE_LOGS} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERY_VIEW} in tenancy
    allow dynamic-group <group_name> to {LOG_ANALYTICS_QUERYJOB_WORK_REQUEST_READ} in tenancy
  3. Additionally, ensure that the user has MANAGE permission on loganalytics-features-family and loganalytics-resources-family. If the user creating the on-demand or scheduled purge has Administrator privileges, then the required permissions are already available:

    allow group <group_name> to MANAGE loganalytics-features-family in tenancy
    allow group <group_name> to MANAGE loganalytics-resources-family in tenancy

For information about dynamic groups and IAM policies, see OCI Documentation: Managing Dynamic Groups and OCI Documentation: Managing Policies.

Archive Log Data

If you're using only the recent logs for your search and analysis tasks in Oracle Logging Analytics, then enable archiving so that you can optimize the storage cost.

Note

  • You can enable archiving only after you have the minimum specified size of data in active storage. Currently, this is 1 TB.

  • The minimum Active Storage Duration (Days) for logs before they can be archived is 30 days.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. Click Enable Archiving. In the Enable Archiving dialog box, enter the count of the days after which the log data in the active storage must be archived in the field Active Storage Duration (Days), and click Enable.

    The count is calculated based on the timestamp of the logs. For example, if your logs have the timestamp November 4, 2020 23:43:12, and you've specified the Active Storage Duration as 30, then the logs will be typically moved to archive storage on December 3, 2020.

    Note

    It must be noted that even if you specify the Active Storage Duration of the logs to determine the logs that must be moved to Archive storage, the log index structure is based on the buckets that are used for storing the logs. In a typical scenario, an entire bucket is moved to the archive storage when all the logs in it are older than the specified criterion.

    For example, consider that the field Active Storage Duration is set to 30 days:

    • Bucket_1 has logs of age 40 - 80 days: The log data is eligible and is moved to archive storage.
    • Bucket_2 has logs of age 25 - 40 days: Although some of the log data is eligible for archiving, it is not archived until all the logs are suitable for the specified age.
    • Bucket_3 has logs of age 0 - 25 days: None of the logs are suitable for archiving. The entire bucket is archived when all the logs become eligible.

    In the above scenario, after Bucket_1 logs are archived, if more logs are collected which are older than 40 days, then they are typically appended to Bucket_2.

  4. If you have enabled archiving already, and want to modify the archiving settings, then click Modify Archiving Settings. You can perform any of the following tasks:

    • You can change the value of the count of the days specified for archiving under Active Storage Duration (Days).
    • Click Disable Archiving to stop archiving.

    Click Save Changes.

Recall Archived Logs

If you want to use the logs that are archived for viewing and analysis, then you can recall the logs. The recalled data will count towards your active storage usage until you release it.

You can recall and release your selected set of logs multiple times. However, the recall feature is enabled only if you already have archived logs.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. In the Storage page, on the left panel under Resources, click Archiving Recall Requests.

    The Archiving Recall Requests page displays the previously initiated recall requests.

  4. Click Create Recall Request. The Create Recall Request dialog box opens.

  5. Select the time range of the logs that you want to recall, by specifying the Start Time and End Time.

  6. Click Estimate Recall Log Size to determine the size of the logs that you've selected for recall.

    Note that the start time and end time are extended to align with the log index structure based on buckets. So, when you view the list of active recalls or visit the activity tab, you may get the start and end time extended beyond your chosen time range.

    If your current recall time specifications overlap with another recall activity, then they can possibly get merged into a single recall activity and the resulting start and end time can get extended.

  7. Click Create to proceed with the recall of the selected logs.

    The recall activity is listed in the Archiving Recall Requests page. The table specifies the status, start time, end time, size, and request date and time of recall activity.

    Watch the status of the recall activity. You can use the recalled logs for viewing and analysis after the recall activity is complete.

  8. After active use of the recalled logs, if you want to release them back to the archive pool, click the menu icon open menu icon in the row corresponding to your recalled logs, and select Release.

    The recalled logs will then be released back into the archive pool. This will enable you to optimize your storage size and cost.

View Storage Activity Report

You can view the summary of your archive, recall, release, and purge activities to maintain close control of your storage use and also to track the status of your key logs that have been part of the activities.

  1. Open the navigation menu and click Observability & Management. Under Logging Analytics, click Administration. The Administration Overview page opens.

  2. The administration resources are listed in the left hand navigation pane under Resources. Click Storage.

    The Storage page is displayed.

  3. In the left panel under Resources, click the Activity Report.

    The page displays the summary of the storage activities initiated such as purge policy, purge on demand, archiving, archiving recall request and recall release.

  4. Use the Activity Type, Status, and Time filters on the left panel to narrow down your search for the storage activities.

  5. Expand the storage activity row to view more details about it.