Publishing Messages

Once a stream is created and active, you can publish messages.

While you can use the Console to publish test messages to a stream after you create it, you need to use PutMessages to populate your stream, or, if you take advantage of the Streaming service's compatibility with Apache Kafka, your Kafka producer can handle message creation.

Messages

Messages are published to a single partition in a stream. If there is more than one partition in the stream, the decision of which partition to publish the message to depends on whether your producers are using the Streaming API and PutMessages, or taking advantage of Streaming's Kafka compatibility and using the Kafka API.

If your producers are using the Streaming API, partitioning is handled server-side by the Streaming service. If your producers are using the Kafka API, partitioning is handled client-side by Kafka.

Server-side partitioning

The partition where a message is published is calculated using the message's key. If the key is null, the partition is calculated using a random 16-byte value. You cannot specify which partition a key uses.

Passing a null key puts the message in a random partition. If a user publishes the same message twice, it could go to different partitions, since a completely new key is generated. Do not expect all messages with a null key to go to the same partition. If you want to ensure that messages with the same value go to the same partition, you should use the same key for those messages.

Handling large messages

If your messages are larger than the 1 MB limit, you can either use chunking or send the message by using Oracle Cloud Infrastructure Object Storage.

  • Chunking: You can split large payloads into multiple, smaller chunks that the Streaming service can accept. The chunks are stored in the service in the same way that ordinary (non-chunked) messages are stored. The only difference is that the consumer must keep the chunks and combine them into the message when all the chunks have been collected. The chunks in the partition can be interwoven with ordinary messages.
  • Object Storage: A large payload is placed in Object Storage and only the pointer to that data is transferred. The receiver recognizes this type of pointer payload, transparently reads the data from Object Storage, and provides it to the end user.

Batching and Throttling

We recommend batching messages to avoid throttling and enable better throughput.

The size of a batch of messages shouldn't exceed 1 MB. If this limit is exceeded, the throttling mechanism is triggered. The throttling mechanism for PutMessages is activated when data write rates exceed 1 MB per second per partition. There is no limitation on the number of writes to a stream, as long as you are under the 1 MB per second per partition throughput.

See Limits on Streaming Resources for more information.

Required IAM Policy

To use Oracle Cloud Infrastructure, you must be granted security access in a policy  by an administrator. This access is required whether you're using the Console or the REST API with an SDK, CLI, or other tool. If you get a message that you don’t have permission or are unauthorized, verify with your administrator what type of access you have and which compartment  you should work in.

For administrators: The policy in Let streaming users manage streams lets the specified group do everything with streaming and related Streaming service resources.

If you're new to policies, see Getting Started with Policies and Common Policies. If you want to dig deeper into writing policies for the Streaming service, see Details for the Streaming Service in the IAM policy reference and Accessing Streaming Resources Across Tenancies.

Using the Console

To publish a message
  1. Open the navigation menu. Under Solutions and Platform, click Analytics, and then click Streaming.
  2. In the list of available streams, click on the stream that you want to publish to.
  3. Click Produce Test Message.
  4. Type the message in the Data text box.
  5. Click Produce.

Using the Command Line Interface (CLI)

For information about using the CLI, see Command Line Interface (CLI). For a complete list of flags and options available for CLI commands, see the Command Line Reference.

To publish a message

oci streaming stream message put --stream-id <stream_id> --messages <JSON_messages> --endpoint <messages_endpoint>
Tip

Provide input for --messages as valid formatted JSON. See Passing Complex Input and Using a JSON File for Complex Input for information about JSON formatting.

For example, file.txt contains the properly formatted JSON. Its values are Base64-encoded:

[
  {
    "key": "a2V5MQ==",
    "value": "dmFsdWUx"
  },
  {
    "key": "a2V5Mg==",
    "value": "dmFsdWUy"
  }
]

The --messages parameter takes the file as its value:

oci streaming stream message put --stream-id ocid1.stream.oc1.phx.exampleuniqueID --messages file://file.txt --endpoint https://cell-1.streaming.us-phoenix-1.oci.oraclecloud.com  
{
  "data": {
    "entries": [
      {
        "error": null,
        "error-message": null,
        "offset": 0,
        "partition": "0",
        "timestamp": "2020-11-03T21:35:03.837000+00:00"
      },
      {
        "error": null,
        "error-message": null,
        "offset": 1,
        "partition": "0",
        "timestamp": "2020-11-03T21:35:03.837000+00:00"
      }
    ],
    "failures": 0
  }
}