Date format configuration

The dateFormats.txt file provides a list of date formats supported by Data Processing workflows. This topic lists the defaults used in this file. You can add or remove a date format from this file if you use the formats supported by it.

If a date in the Hive table is stored with a DATE data type, then it is assumed to be a valid date format and is not checked against the date formats in the dateFormats.txt file. Hive TIMESTAMP values are also assumed to be valid dates, and are also not checked against the dateFormats.txt formats.

However, if a date is stored in the Hive table within a column of type STRING, then Data Processing uses the dateFormats.txt to check if this date format is supported. Both dates and timestamps are then ingested into the Big Data Discovery as Dgraph mdex:dateTime data types.

The dateFormats.txt file is located in the $BDD_HOME/workflowmanager/dp/config directory.

Note: The DP CLI configuration directory ($BDD_HOME/dataprocessing/edp_cli/config) has an identically-named file. However, only the file in the $BDD_HOME/workflowmanager/dp/config directory is used by the Workflow Manager. Therefore, make sure to modify the Workflow Manager file to change the supported date formats.

Default date formats

The default date formats that are supported and listed in the dateFormats.txt file are:

d/M/yy
d-M-yy
d.M.yy
M/d/yy
M-d-yy
M.d.yy
yy/M/d
yy-M-d
yy.M.d
MMM d, yyyy
EEE, MMM d, yyyy
yyyy-MM-dd HH:mm:ss
yyyy-MM-dd h:mm:ss a
yyyy-MM-dd'T'HH-mm-ssZ
yyyy-MM-dd'T'HH:mm:ss'Z'
yyyy-MM-dd'T'HH:mm:ss.SSS'Z'
yyyy-MM-dd HH:mm:ss.SSS
yyyy-MM-dd'T'HH:mm:ss.SSS
EEE d MMM yyyy HH:mm:ss Z
H:mm
h:mm a
H:mm:ss
h:mm:ss a
HH:mm:ss.SSS'Z'
d/M/yy HH:mm:ss
d/M/yy h:mm:ss a
d-M-yy HH:mm:ss
d-M-yy h:mm:ss a
d.M.yy HH:mm:ss
d.M.yy h:mm:ss a
M/d/yy HH:mm:ss
M/d/yy h:mm:ss a
M-d-yy HH:mm:ss
M-d-yy h:mm:ss a
M.d.yy HH:mm:ss
M.d.yy h:mm:ss a
yy/M/d HH:mm:ss
yy/M/d h:mm:ss a
yy.M.d HH:mm:ss
yy.M.d h:mm:ss a

For details on interpreting these formats, see http://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html

Modifying the dateFormats file

You can remove a date format from the file. If you remove a data format, Data Processing workflows will no longer support it.

You can also add date formats, as long as they conform to the formats in the SimpleDateFormat class. This class is described in the Web page accessed by the URL link listed above. Note that US is used as the locale.