Managing Service Connectors

This section describes how to manage service connectors .

A service connector defines the flow of data between a source and target service.

Prerequisites

IAM policies: To use Service Connector Hub, you must be given the required type of access in a policy  written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool.

To move data, your service connector must have authorization to access the specified resources in the source , task , and target  services. Some resources are accessible without policies.

Default policies providing the required authorization are offered when you use the Console to define a service connector. These policies are limited to the context of the service connector. You can either accept the default policies or ensure that you have the proper authorizations in group-based policies.

For more information about service connector authorization, see Access to Source, Task, and Target Services.

If you get a response that you don’t have permission or are unauthorized, check with your administrator. You may not have the required type of access in the current compartment . For more information on user authorizations, see Authentication and Authorization.

Using the Console

To create a service connector
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose a compartment you have permission to work in (on the left side of the page). The page updates to display only the resources in that compartment. If you're not sure which compartment to use, contact an administrator.

  3. Click Create Service Connector.
  4. On the Create Service Connector page, fill in the settings:

    • Connector Name: User-friendly name for the new service connector. Avoid entering confidential information.

    • Description: Optional identifier.
    • Resource Compartment: The compartment where you want to store the new service connector.
    • Configure Service Connector:
      • Source: Select the service containing the data you want to transfer from the following options.
      • Target: Select the service that you want to transfer the data to.
        • Functions: Send data to a function.
        • Logging Analytics: Send data to a log group.
        • Monitoring: Send metric data points to the Monitoring service.
        • Notifications: Send data to a topic.
        • Object Storage: Send data to a bucket.
        • Streaming: Send data to a stream.
      • Configure your source and task:

        Note

        By default, this page uses Basic mode. To toggle between Basic Mode and Advanced Mode, click Switch to Advanced mode (to the right of Configure source connection) or Switch to Basic mode(to the right of Configure source and task). Complex queries cannot be displayed in Basic mode. See I can't view my query in Basic mode and Logging Query Language Specification.

        • Configure source connection:
          Source service Fields
          Logging

          See Logging Overview.

          Compartment

          Log Group

          Logs

          Streaming

          See Streaming Service Overview.

          Note

          This source supports the following targets: Functions, Notifications, Object Storage, and Streaming.

          Compartment

          Stream Pool

          Stream

          Show Advanced Options: Read Position: Specify the cursor position from which to start reading the stream. For more information, see Using Cursors.

          • Latest: Starts reading at messages published after saving the service connector.
          • Trim_Horizon: Starts reading at the oldest available message in the stream.
          Note

          • For stream input schema, see Message Reference.
          • Notifications target with Streaming source: All messages are sent as raw JSON blobs.
        • Configure task:
          Task Fields
          Log Filter Task (filters source logs using the Logging service)
          Audit Logs: When _Audit is selected for Log Group:
          • When Attribute is selected for Filter Type:
            • Filter Type: Attribute
            • Attribute Name
            • Attribute Values
          • When Event type is selected for Filter Type:
            • Filter Type: Event type
            • Service Name
            • Event Type

          Service Logs, Custom Logs: When another log group (not _Audit) is selected for Log Group:

          • Property
          • Operator
          • Value

          Configure Function Task (processes data from the source using the Functions service)

          Note

          This task supports the following targets: Functions, Notifications, Object Storage, and Streaming.

          Function Application: Select the name of the function application that includes the function you want.

          Function: Select the name of the function you want to use to process the data received from the source.

          Show Advanced Options: Optimal batch size: Specify limits for each batch of data sent to the function.

          • Use Automatic Settings
          • Use Manual Settings
            • Batch size limit (KBs)
            • Batch time limit (seconds)
          Note

          • Service Connector Hub does not parse the output of the function task. The output of the function task is written as-is to the target. For example, when using a Notifications target with a function task, all messages are sent as raw JSON blobs.
          • Functions are invoked synchronously with 6MB of data per invocation. If data exceeds 6MB, then the service connector invokes the function again to move the over-limit data. Such over-limit invocations are handled sequentially.
          • Functions can execute for up to five minutes.
        • Click Advanced Mode or Switch to Advanced Mode to view and edit the source connector and task using the Query Code Editor.

    • Configure target connection: Select the Service Compartment (where the target service resides) and fill in additional fields as needed:
      Target service Additional fields
      Functions

      Function Application: Select the name of the function application that includes the function you want.

      Function: Select the name of the function you want to send the data to.

      Note

      • Functions are invoked synchronously with 6MB of data per invocation. If data exceeds 6MB, then the service connector invokes the function again to move the over-limit data. Such over-limit invocations are handled sequentially.
      • Functions can execute for up to five minutes.
      • Do not return data from Functions targets to service connectors. Service Connector Hub does not read data returned from Functions targets.
      Logging Analytics

      Log Group: Select the log group you want.

      Monitoring
      Note

      Namespace: Select the metric namespace that includes the metric you want. It can be an existing or new namespace.

      Metric Name: Select the name of the metric that you want to send the data to. It can be an existing or new metric.

      What's included with your metric
      The following dimensions are included with your metric:
      connectorId
      The OCID of the service connector that the metrics apply to.
      connectorName
      The name of the service connector that the metrics apply to.
      connectorSourceType
      The source service that the metrics apply to.

      The timestamp of each metric data point is the timestamp of the corresponding log message.

      Notifications
      Note

      Log Group for Notifications is limited to _Audit.

      Topic: Select the name of the topic that you want to send the data to.

      Note

      SMS messages exhibit unexpected results for certain service connector configurations. This issue is limited to topics that contain SMS subscriptions for the indicated service connector configurations. For more information, see Multiple SMS Messages for a Single Notification.

      Message Format: Select the option you want:

      Note

      Message Format is available for service connectors with Logging source only. Not available for service connectors with function tasks. When this option is not available, messages are sent as raw JSON blobs.

      • Send formatted messages: Simplified, user-friendly layout.
        Note

        To view supported subscription protocols and message types for formatted messages, see Friendly Formatting.
      • Send raw messages: Raw JSON blob.
      Object Storage

      Bucket: Select the name of the bucket that you want to send the data to.

      Note

      Batch rollover details:
      • Batch rollover size: 100 MB
      • Batch rollover time: 7 minutes

      Files saved to Object Storage are compressed using gzip.

      Streaming

      Stream: Select the name of the stream that you want to send the data to.

    • Show Advanced OptionsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure whether to apply tags, skip this option (you can apply tags later) or ask your administrator.

    Default policies are offered for any authorization required for this service connector to access source, task, and target services.

    You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization.

  5. To accept default policies, click the Create link provided for each default policy.

    Note

    If you don't have permissions to accept default policies, contact your administrator.

    View links are provided for you to optionally review the newly created policies.

  6. Click Create to create the service connector.

    Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.

To edit a service connector
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to edit.
  4. Click Edit.
  5. Make your changes.

    Note

    If you did not previously create the default access policy to allow this service connector to write to the target service, you can do so now. You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization.
  6. Click Save Changes.

    If you updated the source service or tasks, then data movement may pause for a few minutes, as indicated by Data Freshness metrics. Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.

To update your service connector to use friendly message formats

Friendly message formats are available with service connectors that use Notifications as target.

  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to edit.
  4. Click Edit.
  5. Under Configure target connection, select the Message Format you want:

    • Send formatted messages: Simplified, user-friendly layout.
      Note

      To view supported subscription protocols and message types for formatted messages, see Friendly Formatting.
    • Send raw messages: Raw JSON blob.
    Note

    If you did not previously create the default access policy to allow this service connector to write to the target service, you can do so now. You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization.
  6. Click Save Changes.

    If you updated the source service or tasks, then data movement may pause for a few minutes, as indicated by Data Freshness metrics. Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.

To activate a service connector
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to activate.
  4. Click Activate and then confirm.

    The service connector immediately begins moving data according to its configuration, applying tasks to data in the source service and then moving the data to the target service.

To deactivate a service connector
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to deactivate.
  4. Click Deactivate and then confirm.

    The service connector stops moving data.

To move a service connector to another compartment
Note

Default policies stop working for moved service connectors. To give a moved service connector the required authorization, edit the service connector using the Console and accept the offered default policy. For more information about service connector authorization, see Access to Source, Task, and Target Services.
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to edit.
  4. Click Move Resource.
  5. Choose the destination compartment from the list.
  6. Click Move Resource.
To delete a service connector
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
  2. Choose the Compartment containing the service connector.
  3. Click the name of the service connector you want to deactivate.
  4. Click Delete and then confirm.

    The service connector stops moving data.

Using the Command Line Interface (CLI)

To list service connectors

Open a command prompt and run oci sch service-connector list to list service connectors in the specified compartment:

oci sch service-connector list --compartment-id <compartment_OCID>
To get a service connector

Open a command prompt and run oci sch service-connector get to get the specified service connector:

oci sch service-connector get --service-connector-id <service_connector_OCID>
To create a service connector

Open a command prompt and run oci sch service-connector create to create a service connector:

oci sch service-connector create --display-name
"<display_name>" --compartment-id <compartment_OCID> --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]
To edit a service connector

Open a command prompt and run oci sch service-connector update to edit a service connector:

oci sch service-connector update --service-connector-id <service_connector_OCID> --display-name
"<display_name>" --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]
To activate a service connector

Open a command prompt and run oci sch service-connector activate to activate the specified service connector:

oci sch service-connector activate --service-connector-id <service_connector_OCID>
To deactivate a service connector

Open a command prompt and run oci sch service-connector deactivate to deactivate the specified service connector:

oci sch service-connector deactivate --service-connector-id <service_connector_OCID>
To move a service connector to another compartment

Open a command prompt and run oci sch service-connector change-compartment to move the service connector to the specified compartment:

oci sch service-connector change-compartment --service-connector-id <service_connector_OCID> --compartment-id <destination_compartment_OCID>
To delete a service connector

Open a command prompt and run oci sch service-connector delete to delete the specified service connector:

oci sch service-connector delete --service-connector-id <service_connector_OCID>

Using the API