4.2.1.1 Creating a File Stream

To create a File stream:

1. On the Catalog page, click Create New Item.
2. Hover the mouse over Stream and select File from the submenu.
3. On the Type Properties screen, enter the following details:
   • Name: Enter a unique name for the stream. This is a mandatory field.
   • Display Name: Enter a display name for the stream. If left blank, the Name field value is copied.
   • Description
   • Tags
   • Stream Type: The selected stream is displayed.
4. Click Next.
5. On the Source Details screen, enter the following details:
   • File: Upload the CSV or JSON sample file to be used.

     Note:

     Use File stream only for POCs and quick prototyping

   • Read whole content: Select this option to read all the records in the file, at once. If you uncheck this option, the engine reads one record at a time.

   • Number of events per batch: Enter the number of records that you want to process per batch. The default value is one, but you can specify the number of records to process in each read. You can use this option only when Read Whole Content is unchecked.

   • Loop: Select this option to process the file in a loop.

   • Data Format: Select CSV or JSON as the data format.

6. Click Next.
7. On the Data Format screen, set the attributes for the selected the data format.
   • For JSON data format:
     • Allow Missing Column Names: Select this option to allow an input stream that has a column undefined in the shape.
     • Array in Multi-lines: Select this option to allow multi-line data formatting.
   • For CSV data format:
     • CSV Predefined Format: Select one of the predefined data format from the drop-down list. For more information, see Predefined CSV Data Formats.
     • First record as header: Select this option to use the first record as the header row.
8. Click Next.
9. On the Shape screen, select one of the methods to define the shape:

   • Infer Shape : Select this option to detect the shape automatically from the input data stream.

   • Select Existing Shape: Select one of the existing shapes from the drop-down list.

   • Manual Shape : Select this option to infer the fields from a stream or file. You can also update the datatype of the fields.

     Note:

     • To retrieve the entire JSON payload, add a new field with path $.
     • To retrieve the content of the array, add a new field with path $[arrayField].

     In both the cases, the value returned is Text.

   • From File: Select this option to infer the shape from a JSON schema file, or a JSON or CSV data file. You can also save the auto-detected shape and use it later.
10. Click Save.