About the Finite State Machine Framework

In an FSM framework, the standardization process is defined as one or more states. In a state, only the input symbols defined for that state are recognized. When one of those symbols is recognized, the following action or transition is based on configurable processing rules. For example, when an input symbol is recognized, it might be preprocessed by removing punctuation, matched against a list of tokens, and then postprocessed by normalizing the input value. Once this has been completed for all input symbols, the standardization engine determines which token is the most likely match.

FSM-based processing includes the following steps:

• Cleansing – The entire input string is modified to make sure it is broken down into its individual components correctly.

• Tokenization – The input string is broken down into its individual components.

• Parsing – The individual field components are processed according to configurable rules. Parsing can include any combination of the following three stages:

  • Preprocessing – Each token is cleansed prior to matching to make the value more uniform.

  • Matching – The cleansed token is matched against patterns or value lists.

  • Postprocessing – The matched token is normalized.

    ---

    Note –

    Several parsing sequences might be performed against one field component in order to best match it with a token. Each sequence is carried out until a match is made.

    ---

• Ambiguity Resolution – Some input strings might match more than one processing rule, so the FSM framework includes a probability-based mechanism for determining the correct state transition.

Using the person data type, for example, first names such as “Bill” and “Will” are normalized to “William”, which is then converted to its phonetic equivalent. Standardization logic is defined in the standardization engine configuration files and in the Master Index Configuration Editor or mefa.xml in a master index project.