Note:

Forward Logs from Oracle Cloud Infrastructure to Rapid7 InsightOps

Introduction

Oracle Cloud Infrastructure (OCI) is a set of complementary cloud services that enable you to build and run a range of applications and services in a highly available hosted environment. OCI provides high performance compute capabilities (as physical hardware instances) and storage capacity in a flexible overlay virtual network that is securely accessible from your on-premises network.

The Oracle Cloud Observability and Manageability platform aims to meet our customers where they are. We understand that they have standardized their operational postures with popular 3rd party Observability tools, and we want to be interoperable with those tools so our customers can continue using the tools they have invested in with OCI.

In this tutorial, we will walk you through how you can forward logs from OCI into Rapid7 InsightOps. Rapid7 InsightOps is an easy-to-use log management and analytics service. It provides world-class search capabilities, enhanced log analysis tooling, and the ability to monitor and query the real-time state of your infrastructure.

Now, let’s take a look at the high level representation of the solution architecture as shown below.

Architecture Diagram

Objectives

Prerequisites

Task 1: Create a log-Webhook to send your data to Rapid7 InsightOps

HTTP POST is a simple way to forward your log messages to Rapid7 InsightOps.

  1. Log in to Rapid7 InsightOps and click Add Data in the top navigation.

  2. Click Webhook.

    Rapid7 InsightOps-Add Data Source

  3. Enter the name of the log and select an existing log set or create a new one.

    Rapid7 InsightOps-Add a log-Webhook

    Note: This will display a URL that you will use to send your log data to. The last part of the URL is the log token, which identifies the log that the data should be sent to.

Task 2: Configure the logs you want to capture

  1. In the OCI Console, navigate to Observability & Management, Logging and Log Groups.

  2. Select your compartment and click Create Log Group and the side panel opens.

  3. Enter *Rapid7_log_group* for the name, and optionally provide a description and tags.

  4. Click Create to set up your new Log Group.

  5. Under Resources, click Logs.

  6. Click Create custom log or Enable service log as desired.

    For example, let’s enable write logs for an object storage bucket by following the steps below:

    1. Click Enable Service Log. A side panel will open.

    2. Select your resource compartment and type Object Storage in the search services field.

    3. Click Enable Logs and choose your object storage bucket name in the resource field.

    4. Select the previously created log group Rapid7_log_group and select Write Access Events in the Log Category field. Optionally, provide a log name, such as Rapid7_bucket_write.

    5. Click Enable to create your new OCI log.

Task 3: Create an Oracle Function for ingesting logs into Rapid7 InsightOps

  1. In the OCI Console, navigate to Developer services and Functions.

  2. Click Create Application. Give your application a name such as Rapid7_App, select VCN, subnets, shape(GENERIC_X86) from the respective drop down lists and click Create.

  3. Set up your Cloud Shell environment by following the steps below:

    1. Launch Cloud Shell and set up fn CLI on Cloud Shell.

    2. Use the context for your region.

      fn list context
fn use context <region-context>

    3. Update the context with the function’s compartment ID.

      fn update context oracle.compartment-id <compartment-id>

    4. Provide a unique repository name prefix to distinguish your function images.

      fn update context registry <region-key>.ocir.io/<tenancy_name>/[repo-name-prefix]

    5. Generate an Auth Token to Enable Login to Oracle Cloud Infrastructure Registry.

    6. Log into the Registry using the Auth Token as your password.

      docker login -u '<tenancyname>/<username>' <region-key>.ocir.io

    7. Verify your setup by listing applications in the compartment.

      fn list apps

Task 4: Create, deploy and invoke your function

  1. It is recommended to create a boilerplate Python function first and replace the auto generated files with the code as below.

    fn init --runtime python rapid7_func

    Note: The fn init command will generate a folder called rapid7_func with 3 files inside; func.py, func.yaml, and requirements.txt.

  2. Switch to the generated directory.

    cd rapid7_func

    • Replace the contents of func.py with the following sample code. You can adjust the code according to your requirements.

      import io
import os
import json
import requests
import logging
from fdk import response

def process(body):
    try:
        data = body.get("data", {})
        source = body.get("source")
        time = body.get("time")
        oracle = body.get("oracle",{})
        type = body.get("type")
        regionID = body.get("regionID")
        origin = "OCI-CLOUD"

        #Get json data, time, type and source information
        payload = {}
        payload.update({"origin":origin})
        payload.update({"time": time})
        payload.update({"type":type})
        payload.update({"regionID":regionID})
        payload.update({"source":source})
        payload.update({"data":data})
        payload.update({"oracle":oracle})

        #Rapid7 endpoint URL with token to call the REST interface.Refer for more info https://docs.rapid7.com/insightops/insightops-webhook
        #This is defined in the func.yaml file.
        rapid7host = os.environ['RAPID7_HOST']

        #Invoke Rapid7 API with the payload. If the payload contains more than one log this will be ingested as once.
        headers = {'Content-type': 'application/json'}
        x = requests.post(rapid7host, data = json.dumps(payload), headers=headers)
        logging.getLogger().info(x.text)

    except (Exception, ValueError) as ex:
        logging.getLogger().error(str(ex))

      This function accepts the logging JSON and triggers the Rapid7 InsightOps endpoint to ingest the logs. For more information on the top-level logging format, see Logging Format Overview.

      When invoked with multiple logs, the function iterates over each log and triggers the Rapid7 InsightOps endpoint to ingest them individually.

      def handler(ctx, data: io.BytesIO=None):
    try:
        body = json.loads(data.getvalue())
        if isinstance(body, list):
            # Batch of CloudEvents format
            for log in body:
                process(log)
        else:
            # Single log
            process(body)
    except (Exception, ValueError) as ex:
        logging.getLogger().error(str(ex))

    • Replace func.yaml content as follows. RAPID7_HOST must be replaced with the URL that we got in the previous step.

      schema_version: 20180708
name: rapid7_func
version: 0.0.1
runtime: python
entrypoint: /python/bin/fdk /function/func.py handler
memory: 1024
timeout: 120
config:
    RAPID7_HOST: https://us2.webhook.logs.insight.rapid7.com/v1/noformat/1a2345b1-1234-1ab2-1a2b-abcd1ef2345g

    • Replace requirements.txt content as follows.

      fdk
datetime
requests
oci

  3. Deploy your function.

    fn -v deploy --app Rapid7_App

  4. You can test the function by invoking it. If no errors are observed, the function is working as expected.

    fn invoke Rapid7_App rapid7_func

Task 5: Set up an OCI Service Connector

  1. In the OCI console, navigate to Observability & Management, Logging and Service Connectors.

  2. Click Create Service Connector to be directed to the Create Service Connector page.

  3. Enter *Rapid7_SC* for the name, optionally provide a description and select your compartment.

  4. Select the Source as Logging and Target as Functions.

  5. Under Configure Source Connection select a Compartment name, Log Group, and Log (The Log Group and Log created in the first step).

  6. If you also want to send Audit Logs, click +Another Log and select the same Compartment while replacing _Audit as your Log Group.

  7. Under Configure target select a Compartment, Function application, and Function (The Function Application and Function created in the previous step).

  8. If you are prompted to create a policy, click Create from the prompt.

  9. Click Create at the bottom to finish creating your Service Connector.

Task 6: Visualize Oracle Cloud Infrastructure Logs in Rapid7 InsightOps Platform products

To access the logs, log in to Rapid7 InsightOps and navigate to the Log Search section.

Rapid7 InsightOps

Note: The logs ingested into Rapid7 InsightOps through the HTTP post option are also visible in Rapid7 InsightIDR.

Rapid7 InsightIDR

Next Steps

This tutorial has demonstrated the process of forwarding Oracle Cloud Infrastructure (OCI) logs to the Rapid7 InsightOps Platform using the Service Connector Hub and Functions. On the Security Information and Event Management (SIEM) side, it’s essential to define dashboards to capture critical metrics and configure alerts to trigger when predefined thresholds are exceeded. Additionally, defining specific queries is crucial for detecting malicious activities and identifying patterns within your OCI tenancy. These actions will further enhance your security posture and enable proactive monitoring of your cloud environment.

Acknowledgments

Author - Chaitanya Chintala (Cloud Security Advisor)

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.