Before-images Streaming

Learn about streaming before-images to your applications using the Oracle NoSQL Database Streams API.

Oracle NoSQL Database supports streaming before-images. Before-images enable applications to compute the change or delta made by write operations to tables in a data store.

The before-image of any write is the table row before it gets updated or deleted by a DML operation. By default, Oracle NoSQL Database does not generate before-images. After before-images are enabled using either the CREATE TABLE or ALTER TABLE DDL statement, you can configure your subscription to include before-images of subscribed tables in addition to after-images.

To explicitly generate before-images during any writes to a table and stream it to subscribers, you must:

  1. Enable before-images for the table using the ENABLE BEFORE IMAGE syntax. For more details, see Enabling Before-images During Table Creation and ALTER TABLE statement topics.
  2. Configure the subscription to stream before-images to the subscriber for the table of interest. For method details, see Using NoSQLSubscriptionConfig.
  3. Implement the subscriber to receive a stream with before-images events. For method details, see Using the StreamOperation Class.

Following table summarizes the content of a before-image for a DML operation on any user table, either local or Multi-Region:

Table 1-1 Before-images for DML Operations

DML operation Before-image content
INSERT NULL
UPDATE row before update
DELETE deleted row

For either an update or a delete operation, the before-image is the row before it is updated or deleted. For insert operations, before-images are empty.

Note:

  1. Enabling/disabling before-images is on a per table basis and can be done anytime during the lifetime of a table.
  2. Queries and DDL operations do not generate before-images.
  3. The streams can start or resume from any arbitrary location if the location is available. Therefore, streams may include events without before-images if the events were created prior to enabling before-images or after disabling them.
  4. A streamed event is self-describable. From any stream event, a subscriber can distinguish between the following cases by implementing the corresponding methods in the subscriber:
    • Before-images are not enabled for an event.
    • Before-images are enabled but the before-image has expired.
    • Before-images are enabled and has not expired, however, the event does not carry the before-image. For example, an INSERT operation.

    For method details , see oracle.kv.pubsub.StreamOperation class summary in the — Java Direct Driver API Reference.

Before-images expiry and its implications

While the before-images are beneficial to track and compute changes, they consume storage, which can incur a billed charge. To limit this, you can define a TTL value to expire the before-images. The before-images TTL values are set during the before-images generation. After the before-images TTL expires, the before-images don't appear in the stream. If you don't define a before-images TTL value while creating/altering the tables, the system will automatically expire the before-images after 24 hours of its generation. To define before-images TTL when you create a table, see Enabling Before-images During Table Creation. To define before-images TTL for an existing table, see ALTER TABLE statement.

It is possible for a table to have both a table TTL and a before-images TTL. These two TTL values work independently of each other.

You can change the before-images TTL value during the lifetime of a table using the ALTER TABLE statement. The before-images that are generated after the TTL update reflect the new expiry. The existing before-images expiry remain unaffected.

For example, if the before-images are already generated with a TTL of five days and you modify the before-images TTL to one day, then the before-images that are newly generated after the TTL update expire in one day. There is no impact to the expiry of existing before-images.

If you update a row multiple times, each update generates its own before-image. When a new before-image is generated, the expiry of the older before-image for the row remains unaffected and is retained in storage until its expiry is reached.

For example, if a row is updated and before-image is generated with a TTL of 5 days and it is subsequently updated, the second update creates a new before-image with expiry of 5 days from the time of the second update. There is no impact to the expiry of the before-image created by the first update. The new before-image from the second update is the selfsame row resulting from the first update.

Note:

Suppose the subscriber is down and the same row gets updated twice within that time, both before-images are streamed with their respective stream operations within their own expiry dates. For example, the first before-image is streamed with the first update, second before-image is streamed with the second update, and so on.
Consider the following implications associated with setting the before-images TTL. Understanding these tradeoffs is extremely useful to arrive at an optimal value for the before-images TTL.
  1. If you set a small TTL value for before-images and the subscriber is down for a period longer than the TTL value, the before-images can expire before the subscriber comes back on-line, resulting in a loss of the before-images from the stream.
  2. If you set a large TTL value for before-images and the same row gets updated multiple times, all the generated before-images for the row remain in the storage until their respective expiry is reached. This can lead to additional storage overhead as compared to a smaller before-image TTL.
  3. If your table TTL value is lesser than the before-images TTL value, it is possible for the data row to have expired, while the row's before-image may still be active. In such cases, the Streams API can still stream the before-image until its expiry.