Managing Service Connectors
This section describes how to manage service connectors .
A service connector defines the flow of data between a source and target service.
Prerequisites
IAM policies: To use Service Connector Hub, you must be given the required type of access in a policy written by an administrator, whether you're using the Console or the REST API with an SDK, CLI, or other tool.
To move data, your service connector must have authorization to access the specified resources in the source , task , and target services. Some resources are accessible without policies.
Default policies providing the required authorization are offered when you use the Console to define a service connector. These policies are limited to the context of the service connector. You can either accept the default policies or ensure that you have the proper authorizations in group-based policies.
For more information about service connector authorization, see Access to Source, Task, and Target Services.
If you get a response that you don’t have permission or are unauthorized, check with your administrator. You may not have the required type of access in the current compartment . For more information on user authorizations, see Authentication and Authorization.
Using the Console
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
-
Choose a compartment you have permission to work in (on the left side of the page). The page updates to display only the resources in that compartment. If you're not sure which compartment to use, contact an administrator.
- Click Create Service Connector.
-
On the Create Service Connector page, fill in the settings:
-
Connector Name: User-friendly name for the new service connector. Avoid entering confidential information.
- Description: Optional identifier.
- Resource Compartment: The compartment where you want to store the new service connector.
- Configure Service Connector:
- Source: Select the service containing the data
you want to transfer from the following options.
- Logging: Transfer log data from the Logging service. See Logging Overview.
- Streaming: Transfer stream data from the Streaming service. See Streaming Service Overview.
- Target: Select the service that you want to transfer the data to.
-
Configure your source and task:
Note
By default, this page uses Basic mode. To toggle between Basic Mode and Advanced Mode, click Switch to Advanced mode (to the right of Configure source connection) or Switch to Basic mode(to the right of Configure source and task). Complex queries cannot be displayed in Basic mode. See I can't view my query in Basic mode and Logging Query Language Specification.
- Configure source connection:
Source service Fields Logging See Logging Overview.
Compartment
Log Group
Logs
Streaming See Streaming Service Overview.
Note
This source supports the following targets: Functions, Notifications, Object Storage, and Streaming.Compartment
Stream Pool
Stream
Show Advanced Options: Read Position: Specify the cursor position from which to start reading the stream. For more information, see Using Cursors.
- Latest: Starts reading at messages published after saving the service connector.
- Trim_Horizon: Starts reading at the oldest available message in the stream.
Note
- For stream input schema, see Message Reference.
- Notifications target with Streaming source: All messages are sent as raw JSON blobs.
- Configure task:
Task Fields Log Filter Task (filters source logs using the Logging service) Audit Logs: When _Audit is selected for Log Group:- When Attribute is
selected for Filter Type:
- Filter Type: Attribute
- Attribute Name
- Attribute Values
- When Event type is
selected for Filter Type:
- Filter Type: Event type
- Service Name
- Event Type
Service Logs, Custom Logs: When another log group (not _Audit) is selected for Log Group:
- Property
- Operator
- Value
Configure Function Task (processes data from the source using the Functions service)
Note
This task supports the following targets: Functions, Notifications, Object Storage, and Streaming.Function Application: Select the name of the function application that includes the function you want.
Function: Select the name of the function you want to use to process the data received from the source.
Show Advanced Options: Optimal batch size: Specify limits for each batch of data sent to the function.
- Use Automatic Settings
- Use Manual Settings
- Batch size limit (KBs)
- Batch time limit (seconds)
Note
- Service Connector Hub does not parse the output of the function task. The output of the function task is written as-is to the target. For example, when using a Notifications target with a function task, all messages are sent as raw JSON blobs.
- Functions are invoked synchronously with 6MB of data per invocation. If data exceeds 6MB, then the service connector invokes the function again to move the over-limit data. Such over-limit invocations are handled sequentially.
- Functions can execute for up to five minutes.
- When Attribute is
selected for Filter Type:
-
Click Advanced Mode or Switch to Advanced Mode to view and edit the source connector and task using the Query Code Editor.
- Configure source connection:
- Source: Select the service containing the data
you want to transfer from the following options.
- Configure target connection: Select the Service
Compartment (where the target service resides) and fill in
additional fields as needed:
Target service Additional fields Functions Function Application: Select the name of the function application that includes the function you want.
Function: Select the name of the function you want to send the data to.
Note
- Functions are invoked synchronously with 6MB of data per invocation. If data exceeds 6MB, then the service connector invokes the function again to move the over-limit data. Such over-limit invocations are handled sequentially.
- Functions can execute for up to five minutes.
- Do not return data from Functions targets to service connectors. Service Connector Hub does not read data returned from Functions targets.
Logging Analytics Log Group: Select the log group you want.
Monitoring Note
- Do not use the reserved
oci_
prefix for new metric namespaces and names. Metrics are not ingested when reserved prefixes are used. See Publishing Custom Metrics and PostMetricData Reference (API). - When typing a new metric namespace or name, press ENTER to submit it.
Namespace: Select the metric namespace that includes the metric you want. It can be an existing or new namespace.
Metric Name: Select the name of the metric that you want to send the data to. It can be an existing or new metric.
What's included with your metricThe following dimensions are included with your metric:- connectorId
- The OCID of the service connector that the metrics apply to.
- connectorName
- The name of the service connector that the metrics apply to.
- connectorSourceType
- The source service that the metrics apply to.
The timestamp of each metric data point is the timestamp of the corresponding log message.
Notifications Note
Log Group for Notifications is limited to _Audit.Topic: Select the name of the topic that you want to send the data to.
Note
SMS messages exhibit unexpected results for certain service connector configurations. This issue is limited to topics that contain SMS subscriptions for the indicated service connector configurations. For more information, see Multiple SMS Messages for a Single Notification.Message Format: Select the option you want:
Note
Message Format is available for service connectors with Logging source only. Not available for service connectors with function tasks. When this option is not available, messages are sent as raw JSON blobs.
- Send formatted messages: Simplified, user-friendly
layout. Note
To view supported subscription protocols and message types for formatted messages, see Friendly Formatting. - Send raw messages: Raw JSON blob.
Object Storage Bucket: Select the name of the bucket that you want to send the data to.
Note
Batch rollover details:- Batch rollover size: 100 MB
- Batch rollover time: 7 minutes
Files saved to Object Storage are compressed using gzip.
Streaming Stream: Select the name of the stream that you want to send the data to.
- Show Advanced Options: If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure whether to apply tags, skip this option (you can apply tags later) or ask your administrator.
Default policies are offered for any authorization required for this service connector to access source, task, and target services.
You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization.
-
-
To accept default policies, click the Create link provided for each default policy.
Note
If you don't have permissions to accept default policies, contact your administrator.View links are provided for you to optionally review the newly created policies.
-
Click Create to create the service connector.
Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to edit.
- Click Edit.
-
Make your changes.
Note
If you did not previously create the default access policy to allow this service connector to write to the target service, you can do so now. You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization. -
Click Save Changes.
If you updated the source service or tasks, then data movement may pause for a few minutes, as indicated by Data Freshness metrics. Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.
Friendly message formats are available with service connectors that use Notifications as target.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to edit.
- Click Edit.
-
Under Configure target connection, select the Message Format you want:
- Send formatted messages: Simplified, user-friendly
layout. Note
To view supported subscription protocols and message types for formatted messages, see Friendly Formatting. - Send raw messages: Raw JSON blob.
Note
If you did not previously create the default access policy to allow this service connector to write to the target service, you can do so now. You can get this authorization through these default policies or through group-based policies. The default policies are offered whenever you use the Console to create or edit a service connector. The only exception is when the exact policy already exists in IAM, in which case the default policy is not offered. For more information about this authorization requirement, see Authentication and Authorization. - Send formatted messages: Simplified, user-friendly
layout.
-
Click Save Changes.
If you updated the source service or tasks, then data movement may pause for a few minutes, as indicated by Data Freshness metrics. Within a few minutes, the service connector begins moving data according to its configuration. The service connector applies tasks to data from the source service and then moves the data to the target service.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to activate.
-
Click Activate and then confirm.
The service connector immediately begins moving data according to its configuration, applying tasks to data in the source service and then moving the data to the target service.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to deactivate.
-
Click Deactivate and then confirm.
The service connector stops moving data.
Default policies stop working for moved service connectors. To give a moved service connector the required authorization, edit the service connector using the Console and accept the offered default policy. For more information about service connector authorization, see Access to Source, Task, and Target Services.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to edit.
- Click Move Resource.
- Choose the destination compartment from the list.
- Click Move Resource.
- Open the navigation menu and click Analytics & AI. Under Messaging, click Service Connector Hub.
- Choose the Compartment containing the service connector.
- Click the name of the service connector you want to deactivate.
-
Click Delete and then confirm.
The service connector stops moving data.
Using the Command Line Interface (CLI)
Open a command prompt and run oci sch service-connector list
to list
service connectors in the specified compartment:
oci sch service-connector list --compartment-id <compartment_OCID>
Open a command prompt and run oci sch service-connector get
to get the
specified service connector:
oci sch service-connector get --service-connector-id <service_connector_OCID>
Open a command prompt and run oci sch service-connector create
to create
a service connector:
oci sch service-connector create --display-name
"<display_name>" --compartment-id <compartment_OCID> --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]
Open a command prompt and run oci sch service-connector update
to edit a
service connector:
oci sch service-connector update --service-connector-id <service_connector_OCID> --display-name
"<display_name>" --source [<source_in_JSON>] --tasks [<tasks_in_JSON>] --target [<targets_in_JSON>]
Open a command prompt and run oci sch service-connector activate
to
activate the specified service connector:
oci sch service-connector activate --service-connector-id <service_connector_OCID>
Open a command prompt and run oci sch service-connector deactivate
to
deactivate the specified service connector:
oci sch service-connector deactivate --service-connector-id <service_connector_OCID>
Open a command prompt and run oci sch service-connector
change-compartment
to move the service connector to the specified
compartment:
oci sch service-connector change-compartment --service-connector-id <service_connector_OCID> --compartment-id <destination_compartment_OCID>
Open a command prompt and run oci sch service-connector delete
to delete
the specified service connector:
oci sch service-connector delete --service-connector-id <service_connector_OCID>
Using the API
For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.
Use these API operations to manage service connectors:
- Activate ServiceConnector
- ChangeServiceConnectorCompartment
- CreateServiceConnector
- DeactivateServiceConnector
- DeleteServiceConnector
- GetServiceConnector
- ListServiceConnectors
- UpdateServiceConnector
Use these API operations to manage work requests: