Oracle Cloud Infrastructure Documentation

Overview of Service Connector Hub

Service Connector Hub is a cloud message bus platform that offers a single pane of glass for describing, executing, and monitoring movement of data between services in Oracle Cloud Infrastructure.

How Service Connector Hub Works

Service Connector Hub orchestrates data movement between services in Oracle Cloud Infrastructure.

Data is moved using service connectors. A service connector specifies the source service that contains the data to be moved, optional tasks, and the target service for delivery of data when tasks are complete. An optional task might be a function task to process data from the source or a log filter task to filter log data from the source.

Speed of Data Movement

The service connector reads data as soon as it is available. Aggregation or buffering might delay delivery to the target service. For example, metric data points need to be aggregated first.

While service connectors run continuously, data is moved sequentially by individual move operations. The amount of data and the speed of each move operation depend on the service connector configuration (source, task, and target) and the relevant service limits. Relevant service limits are determined by the services selected for the source, task, and target, in addition to limits for Service Connector Hub.

Example 1: Logging source, Notifications target (no task)

Example 1: Logging source, Notifications target (no task).
Callouts for Example 1
Number Description
1 Service Connector Hub reads log data from Logging.
2 Service Connector Hub writes the log data to the Notifications target service.
3 Notifications sends messages to all subscriptions in the configured topic.

Each move operation moves data from the log sources to the topic, within service limits, at a speed affected by the types of subscriptions in the selected topic. The time required for a single move operation to move log sources in this scenario is up to a few minutes.

Example 2: Streaming source, Functions task, Object Storage target

Example 2: Streaming source, Functions task, Object Storage target.
Callouts for Example 2
Number Description
1 Service Connector Hub reads stream data from Streaming.
2 Service Connector Hub triggers theFunctions task for custom processing of stream data.
3 The task returns processed data to Service Connector Hub.
4 Service Connector Hub writes the stream data to the Object Storage target service.
5 Object Storage writes the stream data to a bucket.

Each move operation moves data from the selected stream to the function task and then to the bucket, within service limits and according to size of each batch. (A batch is a list of entries received from the source or task service.) Batch size is configured in task and target settings for this scenario. Once Service Connector Hub receives the stream data from the Streaming service, a single move operation moves a batch of this stream data to the function task according to the task’s batch size configuration and finally moves the processed batch to the bucket according to the target’s batch size configuration. The time required to receive, process, and move a stream in this scenario is up to 17 minutes depending on the task and target batch size configurations.

Service Connector Hub Concepts

The following concepts are essential to working with Service Connector Hub.

service connector

The definition of the data to be moved. A service connector specifies a source service, target service, and optional tasks.

source

The service that contains the data to be moved according to specified tasks—for example, Logging.

target

The service that receives data from the source, according to specified tasks. A given target service processes, stores, or delivers received data—the Functions service processes the received data; the Logging Analytics, Monitoring, Object Storage, and Streaming services store the data; and the Notifications service delivers the data.

task

Optional filtering to apply to the data before moving it from the source service to the target service.

trigger

The condition that must be met for a service connector to run. Currently, the trigger is continuous; that is, service connectors run continuously.

Flow of Data

When a service connector runs, it receives data from the source service, completes optional tasks on the data (such as filtering), and then moves the data to the target service.

Following are the supported targets and optional tasks for each available source, along with a description of the targets.

For examples, see Service Connector Hub Scenarios.

Logging Source

Select a Logging source to transfer log data from the Logging service.

For examples of service connectors using Logging sources, see Scenario: Create Dimensions for a Monitoring Target and other scenarios at Service Connector Hub Scenarios.

All targets are supported by a service connector that is defined with a Logging source and optional task (Functions or Logging).

This image shows the targets supported by a service connector that is defined with a Logging source and optional task.
Callouts for Logging source
Number Description
1 Service Connector Hub reads log data from Logging.
2 Optional: If configured, Service Connector Hub triggers one of the following tasks:
  • Functions task for custom processing of log data.
  • Log Filter task (Logging service) for filtering log data.
3 The task returns processed data to Service Connector Hub.
4 Service Connector Hub writes the log data to a target service.

Example of a service connector that uses Logging as source, with Functions as task: Scenario: Send Log Data to an Autonomous Database.

Monitoring Source

Select a Monitoring source to transfer metric data points from the Monitoring service.

For an example of a service connector using a Monitoring source, see Scenario: Send Metrics to Object Storage.

The following targets are supported by a service connector that is defined with a Monitoring source and (optional) Functions task: Functions, Object Storage, and Streaming.

This image shows the targets supported by a service connector that is defined with a Monitoring source and optional task.
Callouts for Monitoring source
Number Description
1 Service Connector Hub reads metric data from Monitoring.
2 Optional: If configured, Service Connector Hub triggers the following task:
  • Functions task for custom processing of metric data.
3 The task returns processed data to Service Connector Hub.
4 Service Connector Hub writes the metric data to a target service.

Streaming Source

Select the Streaming source to transfer stream data from the Streaming service.

The following targets are supported by a service connector that is defined with a Streaming source and (optional) Functions task: Functions, Notifications*, Object Storage, and Streaming. The Notifications target (asterisked in illustration) is supported except when using the Functions task.

This image shows the targets supported by a service connector that is defined with a Streaming source and optional task.
Callouts for Streaming source
Number Description
1 Service Connector Hub reads stream data from Streaming.
2 Optional: If configured, Service Connector Hub triggers the following task:
  • Functions task for custom processing of stream data.
3 The task returns processed data to Service Connector Hub.
4 Service Connector Hub writes the stream data to a target service.

Targets

Learn when to use each available target.

  • Functions: Send data to a function.
  • Logging Analytics: Send data to a log group.
  • Monitoring: Send metric data points to the Monitoring service.
  • Notifications: Send data to a topic.
  • Object Storage: Send data to a bucket.
  • Streaming: Send data to a stream.

Availability

The Service Connector Hub service is available in all Oracle Cloud Infrastructure commercial regions. See About Regions and Availability Domains for the list of available regions, along with associated locations, region identifiers, region keys, and availability domains.

Resource Identifiers

Most types of Oracle Cloud Infrastructure resources have a unique, Oracle-assigned identifier called an Oracle Cloud ID (OCID). For information about the OCID format and other ways to identify your resources, see Resource Identifiers.

Ways to Access Service Connector Hub

You can access the Service Connector Hub service using the Console (a browser-based interface) or the Service Connector Hub REST API. Instructions for the Console and API are included in topics throughout this guide. For a list of available SDKs, see Software Development Kits and Command Line Interface.

Console: To access Service Connector Hub using the Console, you must use a supported browser. To go to the Console sign-in page, open the navigation menu at the top of this page and click Infrastructure Console. You are prompted to enter your cloud tenant, your user name, and your password. Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.

You can also access Service Connector Hub from the following services in the Console:

  • Logging: Open the navigation menu and click Observability & Management. Under Logging, click Service Connectors.
  • Streaming: Open the navigation menu and click Analytics & AI. Under Messaging, click Streaming. On the stream list page, under Analytics, click Service Connectors.

API: To access Service Connector Hub through API, use Service Connector Hub API.

CLI: See Command Line Reference for Service Connector Hub.

Authentication and Authorization

Each service in Oracle Cloud Infrastructure integrates with IAM for authentication and authorization, for all interfaces (the Console, SDK or CLI, and REST API).

An administrator in your organization needs to set up groups , compartments , and policies  that control which users can access which services, which resources, and the type of access. For example, the policies control who can create new users, create and manage the cloud network, launch instances, create buckets, download objects, etc. For more information, see Getting Started with Policies. For specific details about writing policies for each of the different services, see Policy Reference.

If you’re a regular user (not an administrator) who needs to use the Oracle Cloud Infrastructure resources that your company owns, contact your administrator to set up a user ID for you. The administrator can confirm which compartment or compartments you should be using.

Access to Service Connector Hub

Administrators: For common policies that give groups access to Service Connector Hub, see Allow a group to manage service connectors.

Access to Source, Task, and Target Services

Note

Ensure that any policy you create complies with your company guidelines. Automatically created policies remain when service connectors are deleted. As a best practice, delete associated policies when deleting the service connector.

To move data, your service connector must have authorization to access the specified resources in the source , task , and target  services. Some resources are accessible without policies.

Default policies providing the required authorization are offered when you use the Console to define a service connector. These policies are limited to the context of the service connector. You can either accept the default policies or ensure that you have the proper authorizations in group-based policies.

Default Policies

This section details the default policies offered when you create or update a service connector in the Console.

Functions (Task or Target)

Applies when the service connector specifies a function task or selects Functions as its target service.

Where this policy is created: The compartment where the function resides. The function is selected for the task or target when you create or edit service connector. 

allow any-user to use fn-function in compartment id <target_function_compartment_ocid> where all {request.principal.type='serviceconnector', request.principal.compartment.id='<serviceconnector_compartment_ocid>'} allow any-user to use fn-invocation in compartment id <target_function_compartment_ocid> where all {request.principal.type='serviceconnector', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to use fn-function in compartment id <target_function_compartment_ocid>
where all {
    request.principal.type='serviceconnector',     
    request.principal.compartment.id='<serviceconnector_compartment_ocid>'
}
allow any-user to use fn-invocation in compartment id <target_function_compartment_ocid>
where all {
    request.principal.type='serviceconnector',     
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}
Logging (Source, Task)

No default policies are offered. To create or edit a service connector that specifies logs for the source or task, you must have read access to the specified logs. For more information, see Required Permissions for Working with Logs and Log Groups.

Logging Analytics (Target)

Applies when the service connector specifies Logging Analytics as its target service.

Where this policy is created: The compartment where the log group resides. The log group is selected or entered for the target when you create or edit a service connector. 

allow any-user to use loganalytics-log-group in compartment id <target_log_group_compartment_OCID> where all {request.principal.type='serviceconnector', target.loganalytics-log-group.id=<log_group_OCID>, request.principal.compartment.id=<serviceconnector_compartment_OCID>}

Following is the policy with line breaks added for clarity.

allow any-user to use loganalytics-log-group in compartment id <target_log_group_compartment_OCID>
where all {
    request.principal.type='serviceconnector', 
    target.loganalytics-log-group.id=<log_group_OCID>, 
    request.principal.compartment.id=<serviceconnector_compartment_OCID>
}
Monitoring (Source)

Applies when the service connector specifies Monitoring as its source service.

Where this policy is created: The compartment where the metric namespace resides. The metric namespace is selected or entered for the target when you create or edit a service connector. 

allow any-user to read metrics in tenancy where all {request.principal.type = 'serviceconnector', request.principal.compartment.id = 'compartment_OCID', target.compartment.id in ('compartment_OCID1', 'compartment_OCID2', 'compartment_OCID3')}

Following is the policy with line breaks added for clarity.

allow any-user to read metrics in tenancy where
  all {
    request.principal.type = 'serviceconnector',
    request.principal.compartment.id = 'compartment_OCID',
    target.compartment.id in ('compartment_OCID1', 'compartment_OCID2', 'compartment_OCID3')
  }
Monitoring (Target)

Applies when the service connector specifies Monitoring as its target service.

Where this policy is created: The compartment where the metric namespace resides. The metric namespace is selected or entered for the target when you create or edit a service connector. 

allow any-user to use metrics in compartment id <target_metric_compartment_OCID> where all {request.principal.type='serviceconnector', target.metrics.namespace='<metric_namespace>', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to use metrics in compartment id <target_metric_compartment_OCID>
where all {
    request.principal.type='serviceconnector',
    target.metrics.namespace='<metric_namespace>',
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}
Notifications (Target)

Applies when the service connector specifies Notifications as its target service.

Where this policy is created: The compartment where the topic resides. The topic is selected for the target when you create or edit service connector. 

allow any-user to use ons-topics in compartment id <target_topic_compartment_OCID> where all {request.principal.type= 'serviceconnector', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to use ons-topics in compartment id <target_topic_compartment_OCID>
where all {
    request.principal.type= 'serviceconnector',
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}
Object Storage (Target)

Applies when the service connector specifies Object Storage as its target service.

Where this policy is created: The compartment where the bucket resides. The bucket is selected for the target when you create or edit service connector. 

allow any-user to manage objects in compartment id <target_bucket_compartment_OCID> where all {request.principal.type='serviceconnector', target.bucket.name='<bucket_name>', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to manage objects in compartment id <target_bucket_compartment_OCID> 
where all {
    request.principal.type='serviceconnector',
    target.bucket.name='<bucket_name>',          
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}
Streaming (Source)

Applies when the service connector specifies Streaming as its source service.

Where this policy is created: The compartment where the stream resides. The stream is selected for the source when you create or edit service connector. 

allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment id <source_stream_compartment_OCID> where all {request.principal.type='serviceconnector', target.stream.id='<stream_OCID>', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to {STREAM_READ, STREAM_CONSUME} in compartment id <source_stream_compartment_OCID>
where all {
    request.principal.type='serviceconnector',
    target.stream.id='<stream_OCID>',
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}
Streaming (Target)

Applies when the service connector specifies Streaming as its target service.

Where this policy is created: The compartment where the stream resides. The stream is selected for the target when you create or edit service connector. 

allow any-user to use stream-push in compartment id <target_stream_compartment_OCID> where all {request.principal.type='serviceconnector', target.stream.id='<stream_OCID>', request.principal.compartment.id='<serviceconnector_compartment_OCID>'}

Following is the policy with line breaks added for clarity.

allow any-user to use stream-push in compartment id <target_stream_compartment_OCID>
where all {
    request.principal.type='serviceconnector',
    target.stream.id='<stream_OCID>',
    request.principal.compartment.id='<serviceconnector_compartment_OCID>'
}

When reviewing group-based policies for required authorization to access a resource (service) in a service connector, reference the default policy offered for that service in that context (see previous section) or see the policy details for the service at Policy Reference.

Note

To accept default policies for an existing service connector, simply edit the service connector. The default policies are offered whenever you create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered.

For troubleshooting information, see Troubleshooting Service Connectors.