Oracle Cloud Infrastructure Documentation

Getting Started by Creating a Cluster

Get started with OCI Streaming with Apache Kafka by creating a Kafka cluster  in a OCI tenancy.

Required IAM Policies

Create a policy using these statements in the compartment where you create the Kafka cluster.

define group kafka_admins as ocid1.group.oc1.<.uniqueID>
admit group kafka_admins to {KAFKA_CLUSTER_INSPECT, KAFKA_CLUSTER_READ, KAFKA_CLUSTER_CREATE, KAFKA_CLUSTER_DELETE, KAFKA_CLUSTER_UPDATE, KAFKA_CLUSTER_CONFIG_READ, KAFKA_CLUSTER_CONFIG_INSPECT, KAFKA_CLUSTER_CONFIG_CREATE, KAFKA_CLUSTER_CONFIG_UPDATE, KAFKA_CLUSTER_CONFIG_DELETE, KAFKA_CLUSTER_MOVE, KAFKA_CLUSTER_CONFIG_MOVE } in compartment <compartment> | tenancy
allow service rawfka to use vnics in compartment <compartment> 
allow service rawfka to use network-security-groups in compartment <compartment>
allow service rawfka to use subnets in compartment <compartment>

Creating a Cluster

You must create the prerequisite resources before creating the Kafka cluster.

  • Open the navigation menu  and select Developer Services. Under Application Integration, select OCI Streaming with Apache Kafka.

    From the Kafka clusters page, select Create cluster.

    1. Cluster settings

    • Cluster name: Enter a unique name for the cluster.
    • Compartment: Select the compartment where you want to create the cluster.
    • Apache Kafka version: Select the version of Apache Kafka to use.
    • Tagging: (Optional) If you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you're not sure whether to apply tags, skip this option or ask an administrator. You can apply tags later.

    2. Broker settings

    • Cluster type: Select the Starter cluster or High availability cluster as the type of cluster you want to create. The minimum number of brokers in the cluster changes depending on the type you select. A starter cluster is used for test or development and has 1 broker as the minimum. A high availability cluster is used for production and has 3 brokers as the minimum.
    • Broker setup: Enter the number of brokers you want in the cluster and the OCPU required per broker. These values are best calculated by sizing the cluster according to the cluster requirements. The Memory per broker is allocated based on the number of OCPU per broker you specified. Each OCPU is allocated 6 GB. The default compute shape used for a broker is the Arm-based processor. If this shape isn't available in the region, then a shape with AMD or Intel processor is used.
    • Storage: Specify the block volume storage that's required per broker. Plan the cluster size and storage according to the business requirements to calculate the block volume storage size for the cluster.

    3. Cluster configuration

    A default configuration is provided depending on the type of cluster you're creating. You can use the default configuration or provide a custom configuration. If you choose to use a custom configuration, we recommend you create the cluster configuration file before creating the Kafka cluster. After a Kafka cluster is created, you can still update the cluster configuration file, but with certain limitations.

    4. Security settings (Optional)

    Provide the certificate for Mutual TLS authentication to use mTLS as the authentication mechanism for the cluster. You can also provide the certificate after the cluster is created. To use SASL/SCRAM authentication, you can skip this step.

    5. Networking

    Select the VCN and subnet you created for the cluster. The cluster brokers are created in the subnet you specify

    6. Review and create

    Review all the details provided and make updates, if any, using the Edit link. Then select Create.

  • Use the oci kafka cluster create command and required parameters to create a Kafka cluster:

    oci kafka cluster create 
--access-subnets '[{"subnets":"<subnet-ocid>"}]' 
--broker-shape '{"nodeCount":<node-count>, "ocpuCount":<ocpu-count>, "storageSizeInGbs":storage-size}'
--cluster-config-id <config-ID> 
--cluster-config-version <config-version> 
--cluster-type <cluster-type> 
--compartment-id <compartment-ocid> | -c <compartment-name> 
--coordination-type <coordination-type> 
--kafka-version <version>
--client-certification-bundle <CA-cert-bundle-for-mTLS>
--display-name <cluster-name>
--freeform-tags <tags-in-json>
--auth <auth-type>
--auth-purpose <auth-purpose>
--cert-bundle <CA-cert-bundle-path-for-ssl>
--config-file <config-file-path>
--connection-timeout <connection-timeout-in-seconds>
--defined-tags <tags-in-json>

    Required Options

    <subnet-ocid>
    The set of subnet OCIDs where the Kafka cluster VNIC's must be created.

    You can specify the subnet OCIDs directly in the command prompt in JSON format.

    Or, you can create a JSON file with the subnet information and pass the file in the CLI command. For example, create a subnets.json file with the subnet details and pass the file in the command as --access-subnets file://subnets.json.
    [
  {
    "subnets": [
      "ocid1.subnet.oc1.<.exampleuniqueID_1>",
      "ocid1.subnet.oc1.<.exampleuniqueID_2>"
    ]
  }
]
    Or, you can set the subnet information in a environment export variable and then pass it in the CLI command. For example, create the export variable SUBNETS and pass the variable in the command as --access-subnets "$SUBNETS"
    export SUBNETS='[{"subnets":["ocid1.subnet.oc1.<.exampleuniqueID_1>", "ocid1.subnet.oc1.<.exampleuniqueID_2>"]}]'
    <broker-shape>
    In JSON format, specify the number of broker nodes to create, the number of OCPUs per node, and the storage size for each node.
    <config-ID>
    OCID of the Kafka cluster configuration resource.
    <config-version>
    Version of the Kafka cluster configuration resource.
    <cluster-type>
    The type of cluster to create. Specify either DEVELOPMENT or PRODUCTION.
    <compartment-ID> | <compartment-name>
    The OCID or name of the compartment where the cluster must be created.
    <coordination-type>
    The Kafka broker coordination type. Depending on the Kafka version, specify either ZOOKEEPER or KRAFT.
    <version>
    Supported version to use for creating the Kafka cluster.

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

  • Run the CreateKafkaCluster operation to create a cluster.

Next Steps

You must configure one of the two types of authentication to access the Kafka cluster using a Kafka client application.

Important

Without setting up one of the two types of authentication, you can't access the Kafka cluster you created. To access the Kafka cluster and perform data operations, you must setup a Kafka client.

After the Kafka cluster is created and authentication configured, you can setup a Kafka client in OCI tenancy and run the Kafka commands to create topics, and produce and consume data.