Note:

Move logs from Oracle Cloud Infrastructure into Sumo Logic

Introduction

The Oracle Cloud Observability and Manageability platform aims to meet our customers where they are. We understand that they have standardized their operational postures with popular third-party observability tools and we want to be interoperable with those tools so our customers can continue using the tools they have invested in with Oracle Cloud Infrastructure (OCI).

In this tutorial, we will walk you through how you can move logs from OCI into Sumo Logic. Sumo Logic is a popular observability tool that provides monitoring and security services that provide full visibility into your applications.

Our solution architecture at a high level is as shown below:

Create a Custom HTTP Source Collector in Sumo Logic

In your Sumo Logic account, you need to create an HTTP custom collector app as described in the steps below.

  1. Click Setup Wizard.

  2. Click Start streaming data to Sumo Logic.

  3. Click Your Custom App.

  4. Click HTTPS Source.

  5. Configure your HTTP source as shown below.

    Note: The HTTP source is a metadata tag, stored with your ingested logs, and is useful when searching and filtering logs later in Sumo Logic. Each log line we are going ingest will start with a timestamp of its event occurrence so select the option Use time zone from log file.

    As you move to the next screen, we get the HTTPS endpoint for our logs to upload from OCI, using a POST HTTP call. Please take a note of this endpoint as we will configure our function to use this endpoint to upload logs to Sumo Logic.

Configure the Logs You Want to Capture

You can set up any logs as input for Service Connector Hub and hence ingest into Sumo Logic. For simplifying this tutorial, we will capture Oracle Cloud Infrastructure (OCI) generated logs for write-events to an arbitrary bucket of your choice.

  1. In the Oracle Cloud Console, click the navigation menu, select Logging, and then select Log Groups.

  2. To create a log group, click Create Log Group.

  3. Select your compartment, add LogGroupForBucketActivity for the name and add a description. Click Create.

  4. Select Logs from the Logging menu. You will see a screen similar to below.

  5. Click Enable service log and enter the following information:

    • Service: Select Object Storage
    • Resource: Choose an arbitrary bucket(for example, BucketForSumoLogic) that you would like observed with the logs.
    • Log Category: Select Write Access Events
    • Log Name: Enter a name for your log, for example, logForBucketActivity.
    • Log Group: Select the LogGroupForBucketActivity log group for the log that you just created in the previous step

  6. Click Enable Log.

    Now every time a object is uploaded to the BucketForSumoLogic bucket,a log entry will be added to the logForBucketActivity log.

Configure Oracle Functions for Ingesting Logs into Sumo Logic

  1. In the Oracle Cloud Console, click the navigation menu and select Solution and Platform. Select Functions under the Developer Services menu.

  2. Click Create Application and enter a name, for example, SumoLogicFnApp.

  3. Once you create the application, click your application name and select Getting Started from the Resources menu.

  4. Launch Cloud Shell.

  5. Use the context for your region.

    fn list context
fn use context us-ashburn-1

  6. Update the context with the function’s compartment ID.

    fn update context oracle.compartment-id <compartment-id>

  7. Update the context with the location of the registry you want to use.

    fn update context registry iad.ocir.io/<tenancy_name>/[YOUR-OCIR-REPO]

    Replace iad with the three-digit region code for your region.

  8. Assuming you have created the Auth Token already, log in to the registry using the Auth Token as your password.

    docker login iad.ocir.io

    Replace iad with the three-digit region code for your region.

    You are prompted for the following information:

    • Username: <tenancyname>/<username>
    • Password: Create a password

    Note: If you are using Oracle Identity Cloud Service, your username is <tenancyname>/oracleidentitycloudservice/<username>.

    Verify your setup by listing applications in the compartment.

    fn list apps

  9. Generate a ‘hello-world’ boilerplate function.

    fn init --runtime python sumologicfn

    The fn init command will generate a folder called SumoLogicfn with three files inside: func.py, func.yaml, and requirements.txt.

    Open func.py and replace the content of the file with the following code.

    1. Import the necessary Python modules, as shown in the following snippet.

      import io
import json
import logging
import os

import requests
from fdk import response

    2. Define a function to parse the log data and invoke the Sumo Logic API to ingest the logs.

      # This method is the entrypoint for your Function invokation 
# aka the method invoked by the OCI Fn platform
# it will receive the list of log entries from OCI as input in the form of bytestream
# the method name will be defined in func.yml
def handler(ctx, data: io.BytesIO = None):
    logger = logging.getLogger()
    logger.info("function start")

    # Sumologic endpoint URL to upload OCI logs to HTTP custom app.
    # this value will be defined defined in func.yaml
    sumologic_endpoint = os.environ['SUMOLOGIC_ENDPOINT']

      For information about the format of the logs generated by the Oracle Cloud Infrastructure Logging service, see Logging Format Overview.

    3. Retrieve the log entries from the Service Connector Hub received by our sumologicfn function as its invocation payload. Loop through these log-entries and log-lines one by one.

      try:
    logentries = json.loads(data.getvalue()) # deserialize the bytesstream input as JSON array
    if not isinstance(logentries, list):
        logger.error('Invalid connector payload. No log queries detected')
        raise

    # Optional...log the input to the function as human readble JSON. 
    # Not to be used in production
    logger.info("json input from SCH")
    logger.info(data.getvalue()) 

    for logEntry in logentries: 
        logger.info("Extracting/Parse log details from the log entry json")
        event_name = logEntry["data"]["requestResourcePath"] + '\t'
        time_of_event = logEntry["time"] + '\t'
        cmpt_name = logEntry["data"]["compartmentName"] + '\t'
        bucket_namespace = logEntry["data"]["namespaceName"] + '\t'
        bucket_name = logEntry["data"]["bucketName"] + '\t'
        request_action = logEntry["data"]["requestAction"]

        log_line = time_of_event + event_name + cmpt_name + \
                    bucket_namespace + bucket_name + request_action

        # Call the Sumologic with the payload and ingest the OCI logs
        headers = {'Content-type': 'text/plain'}
        response_from_sumologic = requests.post(sumologic_endpoint,
                                                data=log_line,
                                                headers=headers)
        logging.getLogger().info(response_from_sumologic.text)

    logger.info("function end")
    return

except Exception as e:
     logger.error("Failure in the function: {}".format(str(e)))
     raise

  10. Replace func.yml contents as follows. Make sure you put the value for your SumoLogic_ENDPOINT that we got in the previous step.

    schema_version: 20180708
name: sumologicfn
version: 0.0.1
runtime: python
entrypoint: /python/bin/fdk /function/func.py handler
memory: 1024
timeout: 120
config:
  SUMOLOGIC_ENDPOINT: [YOUR SUMOLOGIC API ENDPOINT URL HERE]

  11. Replace requirements.txt contents as follows.

    fdk
requests

  12. Deploy your function.

    fn -v deploy --app sumologicFnApp --no-bump

  13. Optionally, you can test your SumoLogicfn function with example input as follows:

    curl -O https://raw.githubusercontent.com/mayur-oci/sumologicfn/main/example.json
fn invoke sumologicFnApp sumologicfn < example.json

Create a Service Connector for Reading Logs from Logging and Triggering the Function

  1. In the Oracle Cloud Console, click the navigation menu, and select Solution and Platform. Select Service Connectors under the Logging menu.

  2. Click Create Connector, and from the Source drop-down list, select Logging and from the Functions drop-down list, select Target.

  3. On Configure Source Connection, select your compartment name, your LogGroupForBucketActivity log group, and your logForBucketActivity logs.

  4. If you want to use audit logs, click +Another log, choose your compartment and add _Audit for Log Group.

  5. If prompted to create a policy for writing to Functions, click Create.

The Service Connector is now set up and will trigger the function to ingest logs into Sumo Logic every time it finds logs in the Logging service.

Visualize Oracle Cloud Infrastructure Logs in Sumo Logic

  1. In Sumo Logic, select the Source - Custom App menu to see logs ingested from Oracle Cloud Infrastructure (OCI) using our SumoLogicfn function.

Troubleshoot

This section shows how you can use a simple email alert to monitor the status of your solution.

For more details, see Overview of Functions.

Create a Topic and a Subscription for the Notification Service

  1. In the Oracle Cloud Console, from the navigation menu in the upper-left corner, select Application Integration, and then select Notifications.

  2. Click Create Topic and create a topic with the name my_function_status.

  3. Choose your topic, click Create Subscription and use the following example:

    • Protocol: Email and add create a subscription with your email.

  4. The subscription will be created in “Pending” status. You will receive a confirmation email and will need to click the link in the email to confirm your email address.

Check Metrics and Create an Alarm Definition from Metrics

  1. From the navigation menu in the upper-left corner, select Developer Services, and then select Functions.

  2. Choose the application and the function that you want to monitor.

  3. From the Metrics page, go to the Functions Errors chart, click Options, and then click Create an Alarm on this Query.

  4. Add a name and under Notification, select Destination service as the notification service, select your_compartment, and then select Topic as my_function_status.

Monitor the Status Service Connector Hub

This section shows how you can use a simple email alert to monitor the status of your Service Connector Hub (SCH).

For more details, refer to Service Connector Hub Overview.

Create a Topic and a Subscription for the Notification Service

  1. From the navigation menu in the upper-left corner, select Application Integration, and then select Notifications.

  2. Click Create Topic and create a topic with my_sch_status name.

  3. Choose your topic, click Create Subscription and use the following example:

    • Protocol: Email and add create a subscription with your email

  4. The subscription will be created in “Pending” status. You will receive a confirmation email and will need to click the link in the email to confirm your email address.

Check Metrics and Create an Alarm Definition from Metrics

  1. From the navigation menu in the upper-left corner, select Logging, and then select Service Connectors.

  2. Choose the connector that you want to monitor and from the Resources list in the left navigation panel, select Metrics.

  3. From the metrics chart that you want to add the alarm to, for example, “Service Connector Hub Errors”, click Options and Create an Alarm on this Query.

  4. Add a name and under Notification, select Destination service as the notification service, select your_compartment, and then select Topic as my_sch_status.

Conclusion

This tutorial showed how Oracle Cloud Infrastructure and Sumo Logic customers can configure a highly scalable solution with low overhead for moving logs from Oracle Cloud Infrastructure Logging to Sumo Logic using Service Connector Hub and Oracle Functions.

Acknowledgements

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.