1.3.9 Text Analysis Processors

Text analysis processors offer advanced tools for understanding and improving data stored in text fields. Typically, this data might need to be analyzed and its contents understood, in order to transform it into a new structure. For example, it may be necessary to transform address data that has been manually entered in loosely structured address fields into a more suitable structure for matching. Or, if you are migrating data from several systems to a new system, it may be that the required structure of the data in the new system is different from your sources.

There are two text analysis processors: the Phrase Profiler, and Parse.

The Phrase Profiler analyzes text fields for their contents, and returns the most common words and phrases in the data.

Parse allows you to construct and use rules to understand the contents in full, to validate the data, and to transform it to a new structure if required.