Creating Stream Pools

Stream pools are logical groupings for streams. Every stream needs to be a member of a stream pool. If you don't create a stream pool, the Streaming service uses a default pool to contain your streams.

The following sections describe how to create a stream pool:

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  to work in.

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

Policies for Private Endpoints

To set up a private endpoint, you must have access to a VCN with a private subnet where DNS resolution is enabled. For general information about policies and permissions to do this, see IAM Policies for Networking. Specifically, you need use permissions for a VNIC, a network security group, if you specify one, and a subnet. For example:

allow user group ServiceWriters to use vnics in compartment ABC
allow user group ServiceWriters to use network-security-groups in compartment ABC
allow user group ServiceWriters to use subnets in compartment XYZ

Policies for Encryption Keys

To use your own encryption key, you must let the Streaming service use a Vault key to encrypt data in streams in this stream pool. For example:

allow service streaming to use keys in compartment ABC where target.key.id = '<key_OCID>'

The preceding policy also requires a companion policy to let Streaming use a key on behalf of a user group to create a stream pool that uses the key for cryptographic purposes. For example:

allow user group StreamWriters to use key-delegate in compartment ABC where target.key.id = '<key_OCID>'

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming service in the IAM policy reference and Accessing Streaming Resources Across Tenancies.

Using the Console

  1. Open the navigation menu and click Analytics & AI. Under Messaging, click Streaming.
  2. Click on Stream Pools on the left side of the screen.

    A list of existing stream pools is displayed.

  3. Click Create Stream Pool to display the Create Stream Pool page.
  4. Enter a name for the stream pool in the Stream Pool Name text box. Avoid entering confidential information.
  5. Select a compartment from the Resource Compartment drop-down list.
  6. In the Configure Stream Pool panel:
    1. Select Endpoint Type: Click Public Endpoint or Private Endpoint, depending on whether you want to restrict traffic to streams in this stream pool to a private endpoint that does not require traffic to traverse the internet.

      To create a private endpoint, you need access to a virtual cloud network (VCN) with a private subnet. Select a VCN with a private subnet where DNS resolution is also enabled, and then select the subnet.

      Optionally, if you want to assign a specific private IP address, you must choose one that belongs to the subnet's CIDR. By default, the Networking service assigns a random private IP address on your behalf and applies no security rules to the stream pool. For more information about VCNs and subnets, see VCNs and Subnets.

      Additionally, you can select an existing Network Security Group from the drop-down list to apply the same set of security rules to each stream in the pool.

    2. Configure Encryption Settings: By default, Encrypt using Oracle-managed keys is selected. If you want to encrypt the data in the streams in this stream pool using your own Vault encryption key, click Encrypt using customer-managed keys. To use the Vault service for your encryption needs, you need access to a vault and key and you need to allow the Streaming service to use the key.
      1. Vault: Choose the vault that contains the master encryption key you want to use from the drop-down list.
      2. Master encryption key: Choose the master encryption key you want to use from the drop-down list.

      For more information, see Securing a Stream. For more information about encryption with a Vault key that you manage, see Overview of Vault and Managing Keys.

  7. If you would like to add tags or intend to use Kafka with this stream pool, click Show Advanced Options.
  8. Add TagsIf you have permissions to create a resource, then you also have permissions to apply free-form tags to that resource. To apply a defined tag, you must have permissions to use the tag namespace. For more information about tagging, see Resource Tags. If you are not sure whether to apply tags, skip this option (you can apply tags later) or ask your administrator.
  9. To use the stream pool with Kafka, select the Auto create topics checkbox and configure your stream settings:
    1. Add a number of hours for the stream's retention period in Default Retention Period (Hours) text box.
    2. Specify the Default Number of Partitions for the stream.
    3. Select the View Kafka settings after the Stream pool is created checkbox to display the Kafka Connection settings for the stream pool when it is created.
  10. Click Create.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

Note

The examples in this section use the full syntax for all parameters, for example --compartment-id. For some parameters, there are shortened versions that you can use instead, like -c. See the CLI online help for instances of a shortened parameter associated with a command.
oci streaming admin stream-pool create --name <stream_pool_name> --compartment-id <compartment_OCID>

For example:

oci streaming admin stream-pool create --name MyStreamPool --compartment-id ocid1.tenancy.oc1..exampleuniqueID
{
  "data": {
    "compartment-id": "ocid1.tenancy.oc1..exampleuniqueID",
    "custom-encryption-key": {
      "key-state": "NONE",
      "kms-key-id": null
    },
    "defined-tags": {},
    "endpoint-fqdn": null,
    "freeform-tags": {},
    "id": "ocid1.streampool.oc1.phx.exampleuniqueID",
    "is-private": false,
    "kafka-settings": {
      "auto-create-topics-enable": false,
      "bootstrap-servers": null,
      "log-retention-hours": 24,
      "num-partitions": 1
    },
    "lifecycle-state": "CREATING",
    "lifecycle-state-details": null,
    "name": "MyStreamPool",
    "private-endpoint-settings": {
      "nsg-ids": null,
      "private-endpoint-ip": null,
      "subnet-id": null
    },
    "time-created": "2020-11-02T23:01:59.429000+00:00"
  },
  "etag": "\"b0066564-4bf4-4e27-9255-9055e69a7808-03668273-b0d5-4b8b-9370-74522c29eb56\""
}
Tip

Provide input for --custom-encryption-key-details, --private-endpoint-details, and --kafka-settings as valid formatted JSON. See Passing Complex Input and Using a JSON File for Complex Input for information about JSON formatting.

Using OCI SDKs

To create a stream pool, use the createStreamPool method of StreamAdminClient.

See the Developer Guide to Streaming for detailed SDK examples.

Using Resource Manager and Terraform

Resource Manager is an Oracle Cloud Infrastructure (OCI) service that allows you to automate the process of provisioning your OCI resources. Using Terraform, Resource Manager helps you install, configure, and manage resources through the "infrastructure-as-code" model.

A Terraform configuration codifies your infrastructure in declarative configuration files. The configuration defines the resources you intend to provision, variables, and specific instructions for provisioning the resources

You can use Resource Manager or the Terraform CLI with the OCI Terraform provider to provision Streaming resources like streams and stream pools.

Stream Pool Resource

You can use the oci_streaming_stream_pool resource to create a stream pool with optional private endpoint and Kafka compatibility settings. Private endpoint settings require a VCN, a subnet, and a network security group. This example Terraform configuration creates those resources as well.

For example:

resource "oci_streaming_stream_pool" "test_stream_pool" {
  #Required
  compartment_id = var.compartment_ocid
  name           = "<stream_pool_name>"

  #Optional
  private_endpoint_settings {
    nsg_ids             = [oci_core_network_security_group.test_nsg.id]
    private_endpoint_ip = "10.0.0.5"
    subnet_id           = oci_core_subnet.test_subnet.id
  }

  kafka_settings {
    #Optional
    auto_create_topics_enable = true
    log_retention_hours       = 24
    num_partitions            = 1
  }
}

resource "oci_core_vcn" "test_vcn" {
  cidr_block     = "10.0.0.0/16"
  compartment_id = var.compartment_ocid
  display_name   = "testvcn"
  dns_label      = "dnslabel"
}

resource "oci_core_subnet" "test_subnet" {
  cidr_block     = "10.0.0.0/24"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.test_vcn.id
}

resource "oci_core_network_security_group" "test_nsg" {
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.test_vcn.id
}

For more information about writing configurations for use with Resource Manager, see Terraform Configurations for Resource Manager and Terraform Configuration.