Creating and Managing Streams

Before publishing messages to a stream, or consuming messages from a stream, you must first create a stream.

When creating a stream, consider your partitioning and security strategies.

You can use the Oracle Cloud Infrastructure (OCI) Console, the Command Line Interface (CLI), the Streaming API, the OCI SDKs, and Resource Manager to create your Streaming resources.

Stream Pools

Stream pools are logical groupings for streams. Every stream needs to be a member of a stream pool. If you don't create a stream pool, the Streaming service uses a default pool to contain your streams.

You can use stream pools to:

  • Organize streams into groups matching your organizational structure or a specific solution
  • Restrict access to a specified virtual cloud network (VCN) inside your tenancy so that streams in the pool are not accessible through the internet
  • Specify whether the data in the pool's streams should be encrypted using your own Vault encryption key or an Oracle-managed key

When you create a stream, you need to specify whether it should become a member of an existing stream pool, or a member of a new, automatically created stream pool. There is no limit to the number of stream pools you can create. See Creating Stream Pools for more information.

Note

Stream names must be unique within a stream pool.

Stream Pools and Apache Kafka

Stream pools serve as the root of a virtual Apache Kafka cluster when you use Kafka with Streaming. All streams within the pool share the same Kafka configuration, encryption, and access control settings. Every action on that virtual cluster is scoped to that stream pool.

You can configure the stream pool to automatically create streams, or Kafka topics, and call KafkaAdminClient::createTopic to create a stream or topic in that stream pool.

For more information, see Using Streaming with Apache Kafka.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  to work in.

For administrators: The policy in Let streaming admins manage streaming resources lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference and Accessing Streaming Resources Across Tenancies.