4.4.1.7 Creating HDFS Target
To create HDFS target:
- On the Catalog page, click Create New Item.
- Hover the mouse over Target and select HDFS from the submenu.
- On the Type Properties screen, enter the following details:
- Name: Enter a unique name for the target. This is a mandatory field.
- Display Name: Enter a display name for the target. If left blank, the Name field value is copied.
- Description
- Tags
- Target Type: The selected target is displayed.
- Click Next.
- On the Target Details screen, enter the following details:
- Connection: Select an HDFS connection from the drop-down list.
- HDFS File: Enter a file name. The file name is appended with current timestamp and the extension, based on the type of storage format.
- HDFS Path: Enter the HDFS location. Provide full access to this location to enable users other than the folder owner, to write to this path.
-
File Roll Interval: Enter the roll-over interval to write a new file. The interval can be in 1000ms, 10s, 1m, or 1.5h format.
-
File Roll Max Size: Enter the roll-over file size to create a new file. The size can be in 1000, 10k, 1m, or 1g format.
-
NFS Path: Enter the local file or NFS path where the files are written first and then uploaded to HDFS.
-
Storage Format: Select a storage format from the drop-down list.
- Click Next.
- On the Data Format screen, enter the shape details, based on the storage format you have selected.
- For FILE:
-
File Format: Select a file format from the drop-down list.
- JSON Delimiter: Enter the JSON delimiter if you have selected the JSON file format.
- Avro Codec: Select a compression codec from the drop-down list. This option is enabled if you have selected the file format as AVRO or AVRO Object Container Format.
-
- For PARQUET:
- PARQUET Compression: Select a compression codec from the drop-down list.
- For ORC:
- ORC Compression: Select a compression codec from the drop-down list.
- For FILE:
- Click Next.
- On the Shape screen, select one of the methods to define the shape:
-
Select Existing Shape: Select one of the existing shapes from the drop-down list.
-
Manual Shape: Select this option to manually infer the fields from a stream or file. You can also add to, or remove fields from, an existing shape. Enter the following details:
- Shape Name: Enter a name for the shape.
- Clear Fields: Click to delete all the fields in the shape.
- Field Name: Add the necessary fields.
- Field Type: Select the field data type from the drop-down list.
-
- Click Save.