The Delimited format reads source records that are organized into rows and columns.
Each row is separated from other rows by a row delimiter character, such as the new-line character, and each column is separated from other columns by a column delimiter character, such as a comma or the tab character. The row and column delimiters must not be present within the data itself. For example, if the column delimiter is a comma, no data in a column can contain a comma.
When the source records are read into the Data Foundry, two mappings occur:
- The column names are mapped to Endeca properties. The record adapter assumes that the first row of the file is a header row, which specifies column names separated by the column delimiter. If the first row of the data is not a header row, the column names must be specified on the Pass Throughs tab of the Record Adapter editor.
- The rest of the rows are assumed to contain data and are mapped to Endeca records. These additional rows must have the same number of column delimiters.
Properties are trimmed as they are read in. White space on the ends of properties (including the space, tab, new-line, and other characters) is removed. However, white space within a property is preserved.
The records in a delimited file must have identical properties, in terms of number and type, although it is possible for a record to have a null value for a property. You can use the "Filter empty properties" checkbox, on the Record Adapter editor's General tab, to tell the record adapter to ignore properties that have null values.
- When picking delimiters, you should be aware of the content of the data. Using a delimiter that appears in the content of the data itself can cause problems.
- For the Delimited format, Forge has a built-in hard limit of 64K per record. While this setting cannot be configured, there are workarounds.
Delimited example
This Delimited format illustration uses the pipe ('|') character as the column delimiter. The column names in the header row are mapped to Endeca properties.
The record adapter for this delimited file looks like this:
If the first row of the data is not the header row, you must use the Record Adapter editor's Pass Throughs tab to specify the column names, as in this example:
The Name field must be HEADER_ROW and the Value field must contain the column names separated by the column delimiter.
Tips When Using Delimited Format
- If a delimited file contains new-line delimiters, the delimiter must be specified exactly or unpredictable results may occur. For example, on UNIX systems, the new line is generally a "\n", while on Windows systems, the new line is a "\r\n" combination.
- If a delimited file contains non-standard delimiters, the file may look different than expected. For example, if a delimited format file has "~" as a row delimiter, the data will be entirely on one line (unless new lines appear within the data). Because delimited files typically have one record per row, the file will appear to have a single record or to be corrupt. In fact, files in this format are valid.
