Scenario: Send Log Data to an Autonomous Database

This topic explains how to send log data from Logging to an Autonomous Database using a function (Functions service).

This scenario involves creating a function and then referencing that function in a service connector (Service Connector Hub)  to process and move log data from Logging to an Autonomous Database.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  to work in.

If you're a member of the Administrators group, you already have the required access to execute this scenario. Otherwise, you need access to Functions.

The workflow for creating the service connector includes a default policy when needed to provide permission for writing to the target service.

Setting Up This Scenario

This scenario involves creating a service connector (Service Connector Hub)  to send log data from Logging to an Autonomous Database (JSON).

Before you can create the service connector, you must set up an Autonomous JSON Database that you want to receive the log data and set up the function to copy that log data.

Autonomous JSON Database setup details:

  • Create the Autonomous JSON Database
  • Copy the ORDS base URL: From the detail page for the Autonomous Database, click Service Console, click Development, and then click Copy URL under RESTful Services and SODA.
  • Create a "logs" collection to store the log data that will be moved by the function and service connector:
    1. Go to the detail page for the Autonomous Database.
    2. Click Tools and then click Open SQL Developer Web.
    3. Log in with the admin user and the password you set when you created the database.
    4. In SQL Developer Web, enter the following command:
      soda create logs
    5. Click Run Statement.

    To query documents in the collection after the service connector copies log data, enter the following command:

    soda get logs -f {}

Function setup details:

Once the database and function are set up, you're ready to create the service connector. Creating the service connector is easy in the Console. Alternatively, you can use the Oracle Cloud Infrastructure CLI or API, which lets you execute the individual operations yourself.

Function code sample

The following code sample is for a function to send log data from the Logging service to an Autonomous Database. For instructions on creating and deploying functions, see Creating and Deploying Functions.

Note

The following code sample is not meant for production workloads. Update it for your production environment.
import io
import json
import logging
import requests

from fdk import response

# soda_insert uses the Autonomous Database REST API to insert JSON documents
def soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries):
    auth=(dbuser, dbpwd)
    sodaurl = ordsbaseurl + dbschema + '/soda/latest/'
    bulkinserturl = sodaurl + 'custom-actions/insert/' + collection + "/"
    headers = {'Content-Type': 'application/json'}
    resp = requests.post(bulkinserturl, auth=auth, headers=headers, data=json.dumps(logentries))
    return resp.json()

def handler(ctx, data: io.BytesIO=None):
    logger = logging.getLogger()
    logger.info("function start")

    # Retrieving the Function configuration values
    try:
        cfg = dict(ctx.Config())
        ordsbaseurl = cfg["ordsbaseurl"]
        dbschema = cfg["dbschema"]
        dbuser = cfg["dbuser"]
        dbpwd = cfg["dbpwd"]
        collection = cfg["collection"]
    except:
        logger.error('Missing configuration keys: ordsbaseurl, dbschema, dbuser, dbpwd and collection')
        raise
    
    # Retrieving the log entries from Service Connector Hub as part of the Function payload
    try:
        logentries = json.loads(data.getvalue())
        if not isinstance(logentries, list):
            raise ValueError
    except:
        logger.error('Invalid payload')
        raise
    
    # The log entries are in a list of dictionaries. We can iterate over the the list of entries and process them.
    # For example, we are going to put the Id of the log entries in the function execution log
    logger.info("Processing the following LogIds:")
    for logentry in logentries:
        logger.info(logentry["oracle"]["logid"])

    # Now, we are inserting the log entries in the JSON Database
    resp = soda_insert(ordsbaseurl, dbschema, dbuser, dbpwd, collection, logentries)
    logger.info(resp)
    if "items" in resp:
        logger.info("Logs are successfully inserted")
        logger.info(json.dumps(resp))
    else:
        raise Exception("Error while inserting logs into the database: " + json.dumps(resp))

    # The function is done, we don't return any response because it would be useless
    logger.info("function end")
    return response.Response(
        ctx, 
        response_data="",
        headers={"Content-Type": "application/json"}
    )

Using the Console

This section walks through creating a service connector using the Console. Your function must be deployed.

For help with troubleshooting, see Troubleshooting Service Connectors.

Create a service connector

This example walks through using the Console to create a service connector. In this example, the service connector moves log data from Logging to an Autonomous Database using the function you created using the function code sample.

  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment where you want to create the service connector.
  3. Click Create Service Connector.
  4. On the Create Service Connector page:

    • Type a Connector Name such as "Send Logs to My Autonomous Database." Avoid entering confidential information.
    • Select the Resource Compartment where you want to store the new service connector.
    • Under Configure Service Connector, select your source and target services to move log data to a metric:
      • Select Source: Logging
      • Select Target: Functions
    • Under Configure source connection, select a Compartment Name, Log Group, and Log.
    • Under Configure target connection, select the Function Application and Function corresponding to the function you created using the function code sample.
  5. If prompted to create a policy (required for access to create or update a service connector), click Create.
  6. Click Create.

Using the CLI

This section walks through creating a service connector using the CLI that moves log data to your function (which then moves the data to an Autonomous Database).

For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.
  • Create a service connector: Open a command prompt and run the oci sch service-connector create command:

    oci sch service-connector create --display-name
    "<display_name>" --compartment-id <compartment_OCID> --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]

For help with troubleshooting, see Troubleshooting Service Connectors.

Using the API

This section walks through creating the service connector using the API.

For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.

Use the following operations:

  • CreateServiceConnector: Create a service connector that moves log data to your function (which then moves the data to an Autonomous Database).

    Example CreateServiceConnector request
    POST /20200909/serviceConnectors
    Host: service-connector-hub.us-phoenix-1.oraclecloud.com
    <authorization and other headers>
    {
      "compartmentId": "<compartment_OCID>",
      "description": "My service connector description",
      "displayName": "My Service Connector",
      "source": {
        "kind": "logging",
        "logSources": [
          {
            "compartmentId": "<compartment_OCID>",
            "logGroupId": "<log_group_OCID>",
            "logId": "<log_OCID>"
          }
        ]
      },
      "target": {
        "compartmentId": "<compartment_OCID>",
        "kind": "functions",
        "functionId": "<function_OCID>"
      },
      "tasks": []
      }
    }

For help with troubleshooting, see Troubleshooting Service Connectors.