Add Data Streams

Data streams are created from the Distribution Service. Log in to the Distribution Service to begin creating a Data Stream process. Here are the steps to create a Data Stream:

1. From the Distribution Service home page, click the plus (+) sign next to Data Streams to open the Add Data Stream wizard.

   
   
   

2. On the Data Stream Information page, enter a Data Stream process name in the Name box and add a description for it. Click Next.

   
   
   

3. On the Source Options page, provide the values for options shown in the following image:

   
   

   

   
   

   • Trail Name: Name of the source trail file.

   • Trail Subdirectory: The path of the subdirectory where trail files are stored.

   • Encoding: This option controls the encoding of records for the data stream. Currently, JSON encoding is supported.

   • Buffer Size: This value controls the memory buffer size used in the data streaming service. The data streaming service will flush the in-memory message queue after its total byte size exceeds the specified value and delivers the records to the client.

   • Quality of Service: This option defines the data stream duplicate suppression and recovery behavior. Three levels of quality of service are supported in the data streaming service.

     • Exactly Once: This mode is the most restrictive mode where the service filters out any duplicate records during recovery of seeing a RESTART OK/ABEND records in the source trails. Clients will not see duplicate records. If the service cannot locate the record with given last processed position, an error occurs.

     • At Least Once: This mode does not suppress duplicate records during recovery or seeing a RESTART OK/ABEND records in the source trail. Clients may see duplicate records in the data stream. Also, if the service cannot locate the record with given last processed position, an error occurs.

     • At Most Once: This mode suppresses duplicate records during recovery or seeing a RESTART OK/ABEND records in the source trail. Clients will not see duplicate records in the data stream. If the service cannot locate the record with given last processed position, it will find the next available record and move forward.

   • CloudEvents Format: The data streaming service supports transmitting the data records in CloudEvents format. By default this format is disabled and can be controlled by enabling the property using the toggle switch, when creating the data streaming channel.

4. On the Filtering Options page, the options for include and exclude filtering rules are available, as shown in the following image:
   
   

   Specify the filtering rule options, as follows:

   • Rule Action: Select Exclude or Include options.

   • Filter Type: The filter type includes the following:

     Object Type: You can select multiple options from the drop-down list including DML, DDL, INSERT, UPDATE, UPSERT, DELETE.

     Object Names: Name of the previously created filtering object.

     Click Add to add the filtering rule to the Data Stream process.

5. Click Create Data Stream. You will be returned to the Distribution Service home page where the Data Stream is listed.

6. Click on created Data Stream to view the YAML document Data Streaming AsyncAPI definitions for the stream.