Kafka Connect Converters

Converters help to change the format of data from one format into another format. Converters are decoupled from connectors to allow reuse of converters between connectors naturally. The Converter used at Source and Sink can take input and output to different set of formats. For Example, at the Source Connector, a Converter can take input from JDBC and convert it to AVRO and send it to Kafka Topic. Similarly, at the Sink Connector, a Converter can take input from Kafka Topic as AVRO and send it to OCS. Notice that the data flows from JDBC to OCS.

AVRO Converter is the most common and recommended Converter. This is because the AVRO format is considered to be more stable.

AVRO format support

You can store data in AVRO format from Kafka Topic into any Sink using AVRO Converter.

Note: This is currently available only in OCS Connector.

The following additional elements in payload can be used to Sink data from Kafka Topic into OCS in AVRO format. Refer OCS Sink Connectorfor details.

{
"format.class":""
"key.Converter":""
"value.Converter":""
"partitioner.class":""
"key.Converter.schema.registry.url":""
"value.Converter.schema.registry.url":""
}

Below is an example OCS Sink Connector with AVRO Converter

POST /connectors HTTPS
Host:
Authorization: Basic Auth (Provide the credentials used when creating the cluster)
Content-Type: application/json
{
    "name": "myocsconnector",
    "config": {
        "format.class": "io.confluent.connect.hdfs.avro.AvroFormat",
        "connector.class": "oracle.oehpcs.connect.swift.SwiftSinkConnector",
        "tasks.max": "1",
        "flush.size": "1",
        "topics": "mytopic",
	"ocs.container": "https://tenantname.storage.oraclecloud.com/v1/Storage-tenantname/mycontainer",
        "ocs.username": "Storageadmin",	
        "ocs.password": "mypassword",
        "ocs.tenant": "Storage-tenantname",
        "storage.class": "oracle.oehpcs.connect.swift.SwiftStorage",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "partitioner.class": "oracle.oehpcs.connect.swift.partitioner.SwiftDefaultPartitioner",
        "key.converter.schema.registry.url": "http://<schemaregistryurl>:<portnumber>",
        "value.converter.schema.registry.url": "http://<schemaregistryurl>:<portnumber>",
    }
}

Parquet format support

To enable Connector to support Parquet format, we need to first produce data in AVRO format to Kafka broker and start an OCS Sink Connector with AVRO Converter and Parquet format.

Note: This is currently available only in OCS Connector.

The following additional elements in payload can be used to Sink data from Kafka Topic into OCS in Parquet format.

"format.class":""

Below is an example OCS Sink Connector for Parquet format through AVRO Converter

POST /connectors HTTPS
Host:
Authorization: Basic Auth (Provide the credentials used when creating the cluster)
Content-Type: application/json
{
    "name": "myocsconnector",
    "config": {
        "format.class": "io.confluent.connect.hdfs.parquet.ParquetFormat",
        "connector.class": "oracle.oehpcs.connect.swift.SwiftSinkConnector",
        "tasks.max": "1",
        "flush.size": "1",
        "topics": "mytopic",
	"ocs.container": "https://tenantname.storage.oraclecloud.com/v1/Storage-tenantname/mycontainer",
        "ocs.username": "Storageadmin",
        "ocs.password": "mypassword",
        "ocs.tenant": "Storage-tenantname",
        "storage.class": "oracle.oehpcs.connect.swift.SwiftStorage",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "partitioner.class": "oracle.oehpcs.connect.swift.partitioner.SwiftDefaultPartitioner",
        "key.converter.schema.registry.url": "http://<schemaregistryurl>:<portnumber>",
        "value.converter.schema.registry.url": "http://<schemaregistryurl>:<portnumber>",
    }
}

Notice that the only difference between the AVRO and Parquet format Converters is the format.class element.