Analyzing and Cleansing Data for Sun Master Index

Simple Frequency Analysis Rules

A simple frequency analysis compares the values of the fields you specify and creates a report that lists each value for those fields along with the number of times each value occurs. You can perform the analysis on a single field or multiple fields, and you can sort the resulting report by any of the fields in the report. Each simple frequency analysis rule is defined within SimpleFrequencyAnalysis tags that include the elements and attributes listed in the following table.

Table 1 Simple Frequency Analysis Rules

Element 

Attribute 

Description 

fields 

 

A list of fields to include in the frequency analysis. 

field 

 

One field definition in the list of fields. 

 

fieldName 

The name of the field. If you defined a variable for the field, the syntax for this attribute is fieldName=”:[var_name]”, where var_name is the name you gave the variable. If you did not define a variable, enter the qualified field name within double quotes. For example, fieldName=“Person.FirstName”.

sortOrder 

 

If defined, a field on which to sort in order of frequency. If multiple fields are defined, all of their frequencies will be sorted in this order. 

 

fieldName 

The name of the field on which to sort. Use the syntax described for fieldName above.

 

increasing 

An indicator of whether to sort in increasing or decreasing order. Specify “true” to sort in increasing order, or specify “false” to sort in decreasing order. 

threshold 

 

If defined, a frequency threshold above which or below which field values will be listed on the report. If multiple fields are defined, the threshold setting applies the combination of the fields. 

 

value 

The frequency threshold. This is a cutoff value to help limit the results of the report. 

 

more 

An indicator of whether the threshold is an upper or lower threshold. Specify “true” to return field values with a frequency greater than or equal to the threshold. Specify “false” to return field values with a frequency less than the threshold.  

topNpattern 

 

If defined, the number of top frequencies to display. For example, you can restrict a report to just the top 10 frequencies of a field. If multiple fields are defined, this setting applies to the combination of fields. 

 

value 

The number of top frequencies to display. 

 

showall 

An indicator of whether to display more than the specified number of frequencies if there are multiple values tied at the lowest frequency to display. Specify “true” to show all values that are tied for the top frequencies. Specify “false” to only display the number of frequencies specified by the value element. If there is a tie, the displayed value is selected randomly.


Example 1 Sample Simple Frequency Analysis Rules

The following sample defines an analysis of unique values for first and last names, and the resulting report displays only those values that occur 25 or more times and only those with the top 10 frequencies. If more than one value has the same frequency at the 10th frequency, all of those values are displayed. The results appear in increasing order of frequency.


<SimpleFrequencyAnalysis>
  <fields>
    <field fieldName=":[fname]"/>
    <field fieldName=":[lname]"
  </field>
  <sortOrder fieldName=":[fname]"/>
  <threshold value="25" more="true"/>
  <topNpatterns value="10" showall="true"/>
</SimpleFrequencyAnalysis>
  

The following sample defines an analysis of social security numbers and lists any duplicate values (that is, values that appear two or more times).


<SimpleFrequencyAnalysis>
  <fields>
    <field fieldName="Person.SSN"/>
  </field>
  <sortOrder fieldName="Person.SSN"/>
  <threshold value="2" more="true"/>
<SimpleFrequencyAnalysis>