The flexible framework of the Master Index Standardization Engine allows you to define new FSM-based variants on existing FSM-based data types so you can standardize different categories of the same type of data. For example, you might need to standardize names from several different countries. Variants are easily incorporated into a master index project and can be made globally available to all projects. Perform the following steps to create a custom variant.
The working directory for custom variants requires a specific structure. At a minimum, the working directory will look similar to the following:
/WorkingDir serviceInstance.xml /resource standardizer.xml |
The resource directory might also contain several normalization and lexicon files.
Create a working directory for the new variant.
In the new working directory, create a resource directory.
Continue to Defining the Service Instance.
The serviceInstance.xml file for each variant defines the name of the variant, the data type it modifies, and additional Java class information.
Create a file named serviceInstance.xml at the top level of your working directory.
You can copy a service instance file from an existing variant in the data type to which you will add the new variant, and then modify it for the new variant.
Define values for the elements and attributes described in Service Instance Definition File.
This example defines a new Spanish variant to the PersonName data type.
<serviceInstance type="PersonName"> <description>Person Name Standardization: Spain</description> <parameter name="dataType" value="PersonName" /> <parameter name="variantType" value="SP" /> <componentManagerFactory class="com.sun.inti.components.component.BeanComponentManagerFactory"> <property name="stylesheetURL" value="classpath:/com/sun/mdm/standardizer/impl/standardizer.xsl"/> <property name="urlSource" > <bean class="com.sun.inti.components.url.ResourceURLSource"> <property name="resourceName" value="standardizer.xml /> </bean> </property> </componentManagerFactory> </serviceInstance> |
The value you enter for the variantType parameter must match the name you want the variant to display in the Standardization folder of the master index project.
Save and close the file.
Continue to Defining the State Model and Processing Rules.
The state model defines how the data is read, tokenized, parsed, and modified during standardization. The state model and processing rules are all defined in the standardizer.xml file.
Before you begin this step, determine the different forms in which the data to be standardized can be presented and how it should be standardized for each form. For example, name data might be in the form “First Name, Last Name, Middle Initial” or in the form “First Name, Middle Name, Last Name”. You need to account for each possibility. Determine each state in the process, and the input and output symbols used by each state. It might be useful to create a finite state machine model, as shown below. The model shows each state, the transitions to and from each state, and the output symbol for each state.
For more information about the FSM model, see FSM Framework Configuration Overview.
In /WorkingDirectory/resource, create a new XML file named standardizer.xml.
You can copy the file from an existing variant in the data type to which you are adding the custom variant. Then you can modify the file for the new variant.
If the data you are processing does not need to be parsed, but only needs to be normalized, define normalization rules in the normalizer section of the file.
For more information, see Data Normalization Definitions and Standardization Processing Rules Reference.
If the data you are processing needs to be parsed and normalized, define the state model in the upper portion of the file.
For information about the state model and the elements that define it, see Standardization State Definitions.
The next several steps use the processing rules described in Standardization Processing Rules Reference. Some of these rules might require that you create normalization and lexicon files.
In the inputSymbols section of the file, define each input symbol along with any processing rules.
For more information, see Input Symbol Definitions.
In the outputSymbols section of the file, define each output symbol along with any processing rules.
For more information, see Output Symbol Definitions.
In the cleanser section of the file, define any cleansing rules that should be performed against the data prior to tokenization.
For more information, see Data Cleansing Definitions.
If you created any rules that reference normalization or lexicon files, continue to Creating Normalization and Lexicon Files.
Lexicon files list the possible values for a field so the standardization engine can quickly and accurately recognize different field components. Normalization files list the nonstandard values that might be found in a field along with the standard version so the standardization engine can present a common form for the data. You need to create a file for each lexicon or normalization file you referenced from standardizer.xml.
For more information about normalization and lexicon files, see Lexicon Files and Normalization Files.
For each normalization file you referenced in standardizer.xml, do the following:
Create a text file in /WorkingDirectory/resource.
Save the file under the name you used to reference it from standardizer.xml.
In the file, enter a list of nonstandard values along with their standardized values, separating the nonstandard value from the standard value with a pipe (|) as shown below.
COR|COURT CRT|COURT CR.|COURT CT|COURT CT.|COURT DR|DRIVE DR.|DRIVE DRV|DRIVE ... |
When you are finished, save and close the file.
For each lexicon file you referenced in standardizer.xml, do the following:
Continue to Packaging and Importing the Variant.
Once you have created all the files for the variant, you need to package them into a ZIP file to be imported into a master index application.
In the working directory, select the folder and file at the top level and add them to a ZIP file.
Name the ZIP file the same name as the variant. This is the value you entered for the variantType parameter in Defining the Service Instance.
The ZIP file structure should be similar to the following. Note that this variant includes several normalization and lexicon files. Your variant might not contain any.
Import the file into a master index application as described in Importing Standardization Data Types and Variants in Configuring Sun Master Indexes .
Each data type variant is configured by a service definition file. Service type files define the fields to be standardized for a data type, and service instance definition files define the variant and Java factory class for the variant. Both files are in XML format.
Element |
Attribute |
Description |
---|---|---|
serviceInstance |
A container element for the description and any parameters for the variant. |
|
type |
The name of the data type to which the variant belongs. |
|
description |
A brief description of the variant, such as “Person Names: Spain”. |
|
parameter |
One parameter for the variant. The default variants contain two parameters, dataType and variantType. The dataType parameter specifies the name of the data type to which the variant belongs. The variantType parameter specifies the name of the variant. For a master index application, these are the names of the nodes that appear under the Standardization Engine node. |
|
name |
The name of the parameter. |
|
value |
The value of the parameter. |
|
componentManagerFactory |
The component manager factory class for the variant. |
|
class |
The name of the component manager factory class. The default class is com.sun.inti.components.component.BeanComponentManagerFactory. |
|
property |
A property of the component manager factory class. The default class has two properties. The stylesheetURL property defines the location of the stylesheet, standardizer.xml. The urlSource property defines the process definition file. Its value is a bean (by default, com.sun.inti.components.url.ResourceURLSource), which has a property called resourceName. The value for this property is standardizer.xml. |
|
name |
The name of the property. |
|
value |
The value for the property. |