Note:

Push Logs to a Private Endpoint Stream via OCI Connector Hub using OCI Functions

Introduction

Oracle Cloud Infrastructure (OCI) Functions is a fully managed, multi-tenant, highly scalable, on-demand, Functions-as-a-Service platform. Use OCI Functions when you want to focus on writing code to meet business needs.

OCI Connector Hub helps to transfer data between services in OCI. OCI Connector Hub does not support the private endpoint stream as a target.

The OCI Streaming service provides a fully managed, scalable, and durable solution for ingesting and consuming high-volume data streams in real-time. Use OCI Streaming for any use case in which data is produced and processed continually and sequentially in a publish-subscribe messaging model.

In this tutorial, the source and target for OCI Connector Hub will be OCI Logging and OCI Functions respectively. OCI Connector Hub will check for new logs, once detected it will transfer the logs to OCI Functions and it will trigger the function. OCI Functions will push those logs to the private endpoint stream.

Architecture

Objectives

Prerequisites

Task 1: Create a function using OCI Functions

Create a function in the same private subnet in which the private stream is created using the following code. Below code is an example for OCI Object Storage logs. In the code, you need to replace stream-ocid with a private endpoint stream Oracle Cloud Identifier (OCID) and message-endpoint with a private stream message endpoint URL. We are extracting only few parameters from logs like bucketName, message and clientIpAddress. For more information, see Creating and Deploying Functions.

Note: This function code is an example of pushing OCI Object Storage logs to a private endpoint stream. You need to customize the Python script according to the requirement with different logs.

#!/bin/bash
# Copyright (c) 2016, 2021, Oracle and/or its affiliates.  All rights reserved.
# This software is dual-licensed to you under the Universal Permissive License (UPL) 1.0 as shown at https://oss.oracle.com/licenses/upl or Apache License 2.0 as shown at http://www.apache.org/licenses/LICENSE-2.0. You may choose either license.

import io
import oci
import logging
import json
import os
from fdk import response
from base64 import b64encode, b64decode

def handler(ctx, data: io.BytesIO = None):
    signer_auth = oci.auth.signers.get_resource_principals_signer()
    streaming_client = oci.streaming.StreamClient(config = {}, service_endpoint = "<message-endpoint>", signer=signer_auth)

    logger = logging.getLogger()
    logger.info("function start")

    try:
      logentries = json.loads(data.getvalue()) # deserialize the bytesstream input as JSON array
      if not isinstance(logentries, list):
          logger.error('Invalid connector payload. No log queries detected')
          raise
      logger.info("json input from SCH")
      logger.info(data.getvalue())

      for logEntry in logentries:
          logger.info("Extracting/Parse log details from the log entry json")
          bucketName = logEntry["data"]["bucketName"]
          message = logEntry["data"]["message"]
          clientIpAddress = logEntry["data"]["clientIpAddress"]
          log_line = bucketName + '\t' + message + '\t' + clientIpAddress
          encoded_message = b64encode(log_line.encode()).decode()
          put_messages_response = streaming_client.put_messages(
            stream_id="<stream-ocid>",
            put_messages_details=oci.streaming.models.PutMessagesDetails(
                messages=[
                    oci.streaming.models.PutMessagesDetailsEntry(value=encoded_message)]))

      logger.info("function end")
      return
    except Exception as e:
      logger.error("Failure in the function: {}".format(str(e)))
      raise

Task 2: Create and Set Up the OCI Connector Hub

  1. Click Create connector.

    Create Connector

  2. Enter the Compartment name and select the Logs that you want to push as source.

    Source

  3. Select the created function as target and click Create.

    Target

Now, OCI Connector Hub will look for new logs in source. Whenever a new log is detected, OCI Connector Hub will send that log to OCI Functions. Function will be triggered and it will push the logs to the private endpoint stream.

Acknowledgments

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.