Managing Kafka Connect Configurations

This topic describes how to work with Kafka Connect configurations. The Streaming API calls these configurations harnesses.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  to work in.

To allow a group to manage Kafka Connect configurations, you need to create the correct policy in your tenancy. For example:

allow group KafkaAdmins to manage connect-harnesses in tenancy

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference and Accessing Streaming Resources Across Tenancies.

Using the Console

To create a Kafka Connect Configuration
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. Click Create Kafka Connect Configuration to display the Create Kafka Connect Configuration page.
  4. Select a compartment in the Compartment drop-down list.
  5. Enter a name for the configuration in the Kafka Connect Configuration Name text box. Avoid entering confidential information.
  6. TagsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure whether to apply tags, skip this option (you can apply tags later) or ask your administrator.
  7. Click Create Kafka Connect Configuration.
  8. The details page for the new Kafka Connect configuration has a read-only text box labeled Kafka Connect Storage Topics.
    1. Click Copy to copy the connector configuration so you can paste it into the connect-distributed.properties file for your Kafka connector. For more information, see the official Kafka Connect documentation.
To delete a Kafka Connect Configuration
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. You can delete a Kafka Connect configuration in two ways:
    1. Click the the Actions menu on the right side of the configuration you want to delete and select Delete Kafka Connect Configuration.
    2. Click the configuration you want to delete. The Kafka Connect Configuration Details page displays. Click Delete.
  4. Confirm when prompted.
To move a Kafka Connect Configuration to a different compartment
  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Streaming.
  2. Click on Kafka Connect Configurations on the left side of the screen.

    A list of existing Kafka Connect configurations is displayed.

  3. You can move a Kafka Connect configuration in two ways:
    1. Click the the Actions menu on the right side of the configuration you want to move and select Move Resource.
    2. Click the configuration you want to move. The Kafka Connect Configuration Details screen displays. Click Move Resource.
  4. In the Move Resource to a Different Compartment dialog box, choose the destination compartment from the drop-down list.
  5. Click Move Resource.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

Note

The examples in this section use the full syntax for all parameters, for example --compartment-id. For some parameters, there are shortened versions that you can use instead, like -c. See the CLI online help for instances of a shortened parameter associated with a command.
To get a list of Kafka Connect configurations

oci streaming admin connect-harness list --compartment-id <compartment_OCID>

For example:

oci streaming admin connect-harness list --compartment-id exampleuniqueID
{
  "data": [
    {
      "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
      "defined-tags": {},
      "freeform-tags": {},
      "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
      "lifecycle-state": "ACTIVE",
      "name": "MyKafkaConnectConfig",
      "time-created": "2020-08-31T17:26:09.640000+00:00"
    }
  ]
}

By default, getting a list of Kafka Connect configurations returns up to the first 10 configurations in the compartment.

To create a Kafka Connect configuration

oci streaming admin connect-harness create --name <kafka_connect_configuration_name> --compartment-id <compartment_OCID>

For example:

oci streaming admin connect-harness create --name MyKafkaConnectConfig --compartment-id exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "CREATING",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-11-04T17:36:44.323000+00:00"
  },
  "etag": "\"48886b1f-333f-4142-a335-ed42e88b37f8-0e7d329d-c6df-47fc-b814-f1237b414281\""
}
To view Kafka Connect configuration details

oci streaming admin connect-harness get --connect-harness-id <kafka_connect_configuration_OCID>

For example:

oci streaming admin connect-harness get --connect-harness-id exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {},
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "ACTIVE",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-08-31T17:26:09.640000+00:00"
  },
  "etag": "\"5cca5a19-ad06-42f4-b11e-3945d1b0ece5-a3cec7b6-7b84-4056-b411-10fd9f4b1763\""
}
To update a Kafka Connect configuration

oci streaming admin connect-harness update --connect-harness-id <kafka_connect_configuration_OCID> --defined-tags <JSON_tags>

For example:

oci streaming admin connect-harness update --connect-harness-id exampleuniqueid --defined-tags "{\"tagNamespace\": {\"tagKey1\": \"tagValue1\",\"tagKey2\": \"tagValue2\"}}"
WARNING: Updates to freeform-tags and defined-tags will replace any existing values. Are you sure you want to continue? [y/N]: y 

Select y and press Enter. The Kafka Connect configuration is updated:

{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "defined-tags": {
      "tagNamespace": {
        "tagKey1": "tagValue1",
        "tagKey2": "tagValue2"
      }
    },
    "freeform-tags": {},
    "id": "ocid1.connectharness.oc1.phx.exampleuniqueID",
    "lifecycle-state": "UPDATING",
    "lifecycle-state-details": null,
    "name": "MyKafkaConnectConfig",
    "time-created": "2020-08-31T17:26:09.640000+00:00"
  },
  "etag": "\"f3b74455-2584-4f20-a8bd-24068b1d0cba-2f66ecee-2e34-4c92-90d6-5c643b2be3fb\""
}
To delete a Kafka Connect configuration
Caution

Kafka Connect configurations are deleted immediately. You cannot recover a deleted configuration.

oci streaming admin connect-harness delete --connect-harness-id <kafka_connect_configuration_OCID>

For example:

oci streaming admin connect-harness delete --connect-harness-id ocid1.connectharness.oc1.phx.exampleuniqueID
Are you sure you want to delete this resource? [y/N]:

Select y and press Enter. The Kafka Connect configuration is deleted with no further prompting.

To move a Kafka Connect configuration to a different compartment

oci streaming admin connect-harness change-compartment --connect-harness-id <kafka_connect_configuration_OCID> --compartment-id <target_compartment_OCID>

For example:

oci streaming admin connect-harness change-compartment --connect-harness-id ocid1.connectharness.oc1.phx.exampleuniqueID --compartment-id ocid1.compartment.oc1..exampleuniqueID
{
  "etag": "\"69aae3a1-e5f0-4289-bbbf-950c0fdddf2e-cbdd76e0-9a97-49be-a747-812ba28248d1\""
}

Using the SDKs

In order to use Kafka Connect with Streaming, you need a Kafka Connect configuration, or Kafka Connect harness. You can retrieve the OCID for a harness when you create a new harness or use an existing one. For more information, see Using Kafka Connect.

Creating a Kafka Connect harness

The following code example shows how to create a Kafka Connect harness using the OCI SDK for Java:

CreateConnectHarnessDetails createConnectHarnessDetails = CreateConnectHarnessDetails.builder()
    .compartmentId(compartment) //compartment where you want to create connect harness
    .name("myConnectHarness") //connect harness name
    .build();
 
CreateConnectHarnessRequest connectHarnessRequest = CreateConnectHarnessRequest.builder()
    .createConnectHarnessDetails(createConnectHarnessDetails)
    .build();
 
CreateConnectHarnessResponse createConnectHarnessResponse = streamAdminClient.createConnectHarness(connectHarnessRequest);
ConnectHarness connectHarness = createConnectHarnessResponse.getConnectHarness();
 
while (connectHarness.getLifecycleState() != ConnectHarness.LifecycleState.Active && connectHarness.getLifecycleState() != ConnectHarness.LifecycleState.Failed) {
    GetConnectHarnessRequest getConnectHarnessRequest = GetConnectHarnessRequest.builder().connectHarnessId(connectHarness.getId()).build();
     connectHarness = streamAdminClient.getConnectHarness(getConnectHarnessRequest).getConnectHarness();
}
Listing Kafka Connect harnesses

The following code example shows how to list Kafka Connect harnesses using the OCI SDK for Java:

ListConnectHarnessesRequest listConnectHarnessesRequest = ListConnectHarnessesRequest.builder()
    .compartmentId(comaprtment) // compartment id to list all the connect harnesses.
    .lifecycleState(ConnectHarnessSummary.LifecycleState.Active)
    .build();
 
ListConnectHarnessesResponse listConnectHarnessesResponse = streamAdminClient.listConnectHarnesses(listConnectHarnessesRequest);
List<ConnectHarnessSummary> items = listConnectHarnessesResponse.getItems();