Example Scenario: Detect Anomalies Using Outliers

Using Oracle Log Analytics, you can:

• Reduce millions of log events into a smaller set of patterns

• Rapidly troubleshoot problems by identifying log records that’re behaving different when compared to the expected behavior and intermittent errors

Intelligent drill-down and pivoting gives you additional insight into the cause of the problem by showing a chronological log of entries preceding and following events of interest.

Learn how to use Oracle Log Analytics to troubleshoot the cause for the drop in the number of rides on the online application RideShare.

Jane, one of your ride operators, notices that between 10 p.m. and 11 p.m., there was a sudden drop in the number of rides that were processed. She contacts your application support department and requests help. As the DevOps administrator, you have to troubleshoot this critical problem, because it affects your business.

1. In the RideShare application dashboard, start by filtering the number of processed rides between 10 p.m. and 11 p.m.

   The dashboard shows a sudden dip in the number of processed rides between 10 p.m. and 11 p.m. To troubleshoot this issue, drill down into the details to find out the problem with the application.

2. Click Open in Data Explorer on the top right corner of the Accepted Rides widget and select Edit to view the log entries for the processed rides in the Oracle Log Analytics Data Explorer.
3. In the Fields panel, click Entity, in the Entity dialog box, select all the three hosts and the three applications, and then click Submit to expand your search to include the hosts on which the applications are running.

   Note that the search returned more than 29,000 log entries. Because it’s difficult to analyze so many log entries, try to look for any patterns in these entries by using the cluster command.

   
   

   
   Description of the illustration number-log-entries.png

   
   
4. In the Search field, enter * | cluster and press Enter.

   The cluster command uses machine learning to group log records together based on how similar they are to each other. See Cluster Command in Oracle Log Analytics Search Language Reference.

   Here, the cluster command reduces the large number of log entries into a small number of patterns.

5. Click the right end of the Count column header to sort the messages in reverse order to see which patterns have a small number of entries.

   After the log entries are sorted in the reverse order of message count, you can see some outlier signatures. Outliers are events that occur rarely. Drill down into an outlier to explore further.

   You can see that a log message has returned a fatal error.

   
   

   
   Description of the illustration outlier-fatal.png

   
   
6. In the Count column of the log message with the fatal error, click 1 to display the relevant record.
7. In the log records section, click the menu icon or right-click the record and select Show Logs +/– 1 Minute to see more context for this outlier entry.
   You can see all the log entries that were generated in that 1-minute context.

   You can see that someone had run the chmod command to change permissions on some files. That’s probably the cause of the problem.

8. To investigate further, you can use the advanced log filter options to set a time interval for displaying the log data from a specific entity type or entity. In the log records section, click the menu icon or right-click the record and select Advanced Log Filter Options.

   The Advanced Log Filter Options dialog box gives you options to filter the logs:

   
   

   
   Description of the illustration advanced_log_filter.png

   
   

   You can use the advanced options to drill down on a specific time range and quick pick options to add as a query filter.

   • All Entities: If you previously selected specific entities, then this option will clear the selection. If a composite entity is selected, then it will be retained. The query is set to * to filter all the logs for all the entities. This will find all the logs for the specified time range.

   • Entity = <entity> | Entity Type = <entity type> | Log Entity = <log entity>: This option includes the specific filters for the selected row including entity, entity type and log entity, and add them to the query. This is similar to the 1-minute, 5-minutes, and 10-minutes contexts, but helps you to set a specific time range.  With this selection, you can find all the logs matching the row selection criteria.

   • Entity = <entity>: This option includes only the entity filter and adds that to the query. With this selection, you can find all the logs for the selected entity.

   • Entity Type = <entity type>: This option includes only the entity type filter and adds that to the query. With this selection, you can find all the logs for a selected entity type.

   • Show Problem Logs Only: Select this to view only the problem logs from the result of the other selections.