The following sections describe administration user interface help for Oracle Enterprise Data Quality:
Describes EDQ administration capabilities.
The Launchpad controls access to the EDQ applications, and global navigation links such as Administration, Dashboard, and Web Services. To access all the pages and options available to you, use the Launchpad's login functionality, by selecting Log In from the top right of the screen.
Once logged in, you can see the full list of navigation links and applications that you have permission to access. If you navigate to a page that requires authentication, you are directed through the log in page.
Once you are logged in you have access to a user menu by clicking on your user name in the top right of the screen. This provides access to log out and change password functionality (a Change Password option is only available if using an internal realm).
In the global navigation Launchpad and Dashboard are both buttons that take you to those pages, whereas Administration and Web Services are menus that allow you to chose from a set of more specific pages. The Web Services menu contains Web Services and Web Service Tester. The Administration menu contains pages for each of the sections Users, Groups, External Groups, Sessions, Extensions and Launchpad Configuration.
Upon entering one of these pages there will be a left hand navigation panel to move between pages within that global navigation section more fluidly, with your current page being highlighted.
To navigate back to the Launchpad screen, select Launchpad.
Describes accessing and configuring the administration options:
This tab controls the applications that are published to the Launchpad. Users must be granted permission to launch any of the applications in the EDQ suite. Application permissions are associated with user groups, and a user can only launch an application if they belong to at least one group that is granted permissions for that application.
The main page displays each of the selected launchers in the position it will appear in on the Launchpad, each with their name, icon, description and help link. To the side there is a list of the available but not currently displayed launchers. Upon pressing the button to configure the Launchpad you are taken to a modal page with a very similar layout, except now there are buttons and drag and drop features available on each launcher depending on its location.
Note:
The Issue Management application can be accessed within Director. It is surfaced as a separate application to support linking into Director from issue notification emails. It can be used by users in the configured groups from email links without publishing it to the Launchpad. The Configuration Analysis, Dashboard and Server Console applications can also be launched from Director. Application and functional permissions are always respected.The Extensions tab is used to manage the extensions that are installed on an EDQ server.
The Functional Packs tab allows you to manage the Oracle Enterprise Data functional packs according to your license agreement with Oracle. You need the "Change Functional Packs" permission to be able to Edit the packs selected.
Oracle allows its customers to evaluate any features on a trial basis, but reserves the right to audit any user to ensure that production systems are in compliance with their license agreement.
The functional packs specify which sets of processors are activated in an installation, whether or not real-time processing is enabled, and whether or not the Dashboard application is enabled.
If a family of processors is not selected, processors in this family will not be available for you to use. If real-time processing is not enabled, it will not be possible to use the Web Service or JMS interfaces. If Dashboard is not enabled, it will not be possible to publish data quality metrics to the Dashboard.
The User tab contains information on EDQ user accounts.
Specifically, it is used to:
Create and administer user accounts, including assigning users to groups and configuring individual security settings.
Configure universal security settings.
The Users tab is not visible in a default installation of EDQ that uses WebLogic Server. By default, WebLogic Server uses Oracle Platform Security Services (OPSS) to manage users, and EDQ reads the groups from WebLogic Server. To manage users in a default WebLogic environment, create them and assign them to groups in WebLogic Server, then map those groups to EDQ groups on the External Groups tab.
Note:
EDQ also supports integration with external user management systems such as Oracle Internet Directory and Active Directory, enabling external management of users and passwords. See the EDQ LDAP Integration Guide.Access rights in EDQ are controlled via groups. Permissions for applications, and project and functional point access are assigned to groups. Users belong to groups, and inherit their permissions from the groups to which they belong.
Several default groups with pre-configured permissions are provided with a new installation of EDQ. These are detailed below. It is also possible to create new groups and to assign permissions to those groups.
Users must be granted permission to launch any of the applications in the EDQ suite. Application permissions are associated with user groups, and a user can only launch an application if they belong to at least one group that is granted permissions for that application.
This tab only appears on EDQ instances that are integrated with an external user management system, such as Oracle Internet Directory or Active Directory or Oracle Platform Security Services.
It lists the available external realms and groups, and the EDQ groups they can be or are mapped to.
The Sessions tab is used to monitor user sessions by sessions and time (days).
The sessions are listed by username, and include each session's start time, end time (left blank if the session is current), and provides links to the Client and Server Logs.
Active sessions are marked with a Active Session indicator to the left of the user name.
The sessions displayed can be filtered using the Filter Results multi-selection area on the left of the screen.
By default, the sessions for all users over the last 30 days are displayed. To filter the results:
Select and move usernames from the Included Users to the Excluded Users list.
Edit the Number of Days field as required (maximum value is 30).
Click Filter to apply the new parameters.
All the Web Services that are set up on an EDQ server can be viewed from a single page so that integrators can easily access the WSDL files required to use them.
The Web Service Tester application is automatically loaded when the service is selected;. For further information, see Web Service Tester.
Note:
Users with restricted project access will only be able to view and test the Web Services for the projects they have access to.EDQ is a fully extensible system, allowing you to create and plug in your own processors and match extensions into the application.
Note that processors are termed 'widgets' internally, which will help to explain some of the terms used in this section.
New processors are created as JAR files consisting of the following files:
widgets.xml (processor definition and optional javascript implementation)
compiled Java class files (optional implementation, if you need to add functionality not covered by Java libraries in the specification of Java 1.6 or later)
widgets.properties (optional text file facilitating internationalization of labels)
image files (optional icon image files for custom processors)
version.properties (mandatory file containing version information)
See http://java.sun.com/docs/books/tutorial/deployment/jar/index.html for information on packaging programs in JAR files.
These JAR files need to be copied to the EDQ oedq_local_home\localwidgets folder. The new processor will be automatically deployed and added to the tool palette when the EDQ Application Server service is restarted on the EDQ server.
To change the location of the configuration directory for an EDQ server, use the Configuration option on the EDQ Launchpad, and log on with an administrator's user name and password.
Upgrading EDQ will always preserve any custom processors. Furthermore, the EDQ uninstallation and upgrade processes create a full backup of a server configuration before proceeding (in [Common Files]).
New processors can mimic almost the full functionality of other EDQ processors, including:
Defining processor options ('properties'), including those that require Reference Data ('resources')
Defining output filters ('spigots') from the processor
Defining how the processor generates results
Transforming data
See the following Example written processor.
The extensibility of matching is a special case, where the match processors themselves are complex processors that could not easily be written from scratch, but where their processing is extensible. This allows you to define your own matching algorithms and use them within the existing EDQ match processors.
The following may all be added (click for more information and examples):
You can also create your own custom output selectors; see the Selection Functions section for more details.
Note that Comparisons and Matching transformations are collectively termed 'gadgets' internally.
Matching is extended via the creation of JAR files in a similar way to adding processors. The JAR files consist of the following files:
matchlibrary.xml (if you want to add new identifier types and default result bands for comparisons)
widgets.xml (if you want to add new gadgets)
matchlibrary.properties (optional text file facilitating internationalization of labels)
widgets.properties (optional text file facilitating internationalization of labels).
These JAR files need to be copied to the oedq_local_home\localgadgets folder. The new matching functionality will be automatically deployed and available within all match processors when the EDQ Application Server service is restarted on the server.
For further examples of custom-created gadgets, and how to create your own, please contact support.
Selection functions are defined using a similar mechanism to widgets and gadgets, using a widgets.xml to define the selection function file and a widgets.xml file to facilitate internationalization.
Selection functions should be uploaded to the oedq_local_home\localselection folder. See the Example custom output selector topic for more details.
Extensions are imported and managed using the Extensions in the EDQ Administration application.
Custom comparisons may be added into the match library - they are added to widgets.xml in the same way as processors (widgets). The only limitation is that a comparison must have exactly two inputs and one output. Outputs must be either strings (for Boolean comparisons) or numbers (for comparisons that use Result Bands). Boolean comparisons return "T" for True or "F" for False.
Each custom comparison must be associated with an identifier type - either an existing type (String, Number or Date), or a custom type - see Example custom identifier type.
Associating comparison gadgets with identifier types
Comparison gadgets must be associated for use with specific Identifier types. If you want to associate new comparisons with existing system Identifiers, their names are:
dnm:string for Strings
dnm:number for Numbers
dnm:date for Dates
The following example xml represents a comparison association added to matchlibrary.xml:
<identifierComparison>
<ident>dnm:string</ident>
<gadget>dnm:exactstringmatch</gadget>
</identifierComparison>
This associates the identifier "dnm:string" with the comparison "dnm:exactstringmatch".
Setting default result bands for comparisons
The following xml represents a comparison default result band added to matchlibrary.xml for the 'String Edit Distance' comparison:
<comparisonReturn>
<widgetId>dnm:stringeditdistance</widgetId>
<resultBand name="exact" label="Exact Match">0</resultBand>
<resultBand name="onetypo" label="One Typo">1</resultBand>
<resultBand name="twotypos" label="Two Typos">2</resultBand>
<resultBand name="threetypos" label="Three Typos">3</resultBand>
</comparisonReturn>
The following example files may be packaged in a JAR file and used to add a custom 'Character Transposition Match' comparison to the match library. The Character Transposition Match comparison matches strings where character transpositions have occurred. For example, when comparing the values 'Michael' and 'Micheal', a single transposition will be counted, so the two values will match if the Maximum allows transpositions option is set to 1 or higher:
<?xml version="1.0" encoding="UTF-8"?>
<!--
Custom Match Library Extension
Copyright 2008 Oracle Ltd. All rights reserved.
-->
<matchLibrary>
<identifierComparison>
<ident>dnm:string</ident>
<gadget>dn:characterTranspositionMatch</gadget>
</identifierComparison>
</matchLibrary>
<?xml version="1.0" encoding="UTF-8"?>
<widgets>
<comment>Oracle Match example script widgets</comment>
<copyright>Copyright 2008 Oracle Ltd. All rights reserved.</copyright>
<widget id="dn:characterTranspositionMatch" class="com.datanomic.director.match.library.util.JavaScriptGadget">
<guidata>
<label>%characterTranspositionMatch.gadget</label>
<group>compare</group>
<icon>script</icon>
</guidata>
<!-- inputs -->
<inputs>
<input id="1" type="string" maxattributes="1">
<guidata><label>label1</label></guidata>
</input>
<input id="2" type="string" maxattributes="1">
<guidata><label>label1</label></guidata>
</input>
</inputs>
<!-- outputs -->
<outputs cardinality="1:1">
<output id="1" type="string" name="result">
<guidata><label>resultlabel</label></guidata>
</output>
</outputs>
<properties>
<property name="matchNoDataPairs" type="boolean" required="true">
<guidata>
<label>%characterTranspositionMatch.property.matchNoDataPairs.label</label>
</guidata>
<default>false</default>
</property>
<property name="ignoreCase" type="boolean" required="true">
<guidata>
<label>%characterTranspositionMatch.property.ignoreCase.label</label>
</guidata>
<default>true</default>
</property>
<property name="startsWith" type="boolean" required="true">
<guidata>
<label>%characterTranspositionMatch.property.startsWith.label</label>
</guidata>
<default>false</default>
</property>
<property name="maxAllowedTranspositions" type="number" required="true">
<guidata> <label>%characterTranspositionMatch.property.maxAllowedTranspositions.label</label>
</guidata>
<default>1</default>
</property>
</properties>
<parameters>
<parameter name="script">
<![CDATA[
function S(s)
{
return (s == null) ? "" : s;
}
function doit()
{
// no data pairs
if (S(input1) == "" | S(input2) == "")
{
if (matchNoDataPairs)
output1 = "T";
else
output1 = "F";
return;
}
if (!startsWith)
{
if (input1.length != input2.length)
{
output1 = "F";
return;
}
}
var transpositions = 0;
var longword = input1.length > input2.length ? input1 : input2;
var shortword = input1.length > input2.length ? input2 : input1;
if (ignoreCase)
{
// convert to uppercase
longword = longword.toUpperCase();
shortword = shortword.toUpperCase();
}
for (var i = 0; i < shortword.length; i++)
{
if (shortword[i] != longword[i])
{
// are we at the end of the string?
if (i == shortword.length - 1)
{
output1 = "F";
return;
}
// not a transposition match?
if (shortword[i] != longword[i + 1])
{
output1 = "F";
return;
}
// compare the next character
if (shortword[i + 1] != longword[i])
{
output1 = "F";
return;
}
transpositions++;
// too many transpositions?
if (transpositions > maxAllowedTranspositions)
{
output1 = "F";
return;
}
// skip over the characters
i++;
}
}
output1 = "T";
}
]]>
</parameter>
<parameter name="function">doit</parameter>
</parameters>
</widget>
</widgets>
Example 2-3 matchlibrary.properties
[This file was not required in this case as the comparison does not support result bands, and does not require new identifiers.]
Example 2-4 widgets.properties
characterTranspositionMatch.gadget = Character Transposition Match characterTranspositionMatch.property.matchNoDataPairs.label = Match No Data pairs? characterTranspositionMatch.property.ignoreCase.label = Ignore case? characterTranspositionMatch.property.startsWith.label = Starts with? characterTranspositionMatch.property.maxAllowedTranspositions.label = Maximum allowed transpositions
The xml describing a custom identifier type uses specific tags, as follows:
<name> denotes the identifier's internal name and must be unique (standard match identifiers are all prefixed with "dnm:")
<label> denotes the identifier's label as it appears in the GUI
<description> denotes the identifier's description as it appears in the GUI
<systemDefinition> is reserved for future use and must be set to false.
Example 2-6 Custom Identifier added to matchlibrary.xml
The following example xml represents a custom identifier added to matchlibrary.xml:
<identifierDefinition>
<name>newidentifier</name>
<label>My New Identifier</label>
<description>The description</description>
<systemDefinition>false</systemDefinition>
<formalAttribute>
<name>value1</name>
<label>First value</label>
<type>string</type>
</formalAttribute>
<formalAttribute>
<name>value2</name>
<label>Second value</label>
<type>number</type>
</formalAttribute>
</identifierDefinition>
See also the Example custom comparison for how to associate comparisons with identifier types.
Custom output selectors are added to a widgets.xml file in the same way as EDQ processors (widgets) and saved in the oedq_local_home\localselection folder.
Example 2-7 Example of a script-based gadget for an Output Selector
The following is an example of a script-based gadget for an Output Selector:
<!-- ************************************************************ -->
<!-- script for a simple 'first value' selection gadget -->
<!-- ************************************************************ -->
<widget id="dnm:customselect"
class="com.datanomic.director.match.library.util.JavaScriptSelectionGadget">
<guidata>
<label>%custom.firstvalue.name</label>
<group>select</group>
<icon>script</icon>
</guidata>
<!-- inputs -->
<inputs>
<input id="1" type="string" maxattributes="unlimited">
<guidata><label>%custom.firstvalue.input</label></guidata>
</input>
</inputs>
<!-- outputs -->
<outputs cardinality="1:1">
<output id="1" type="string" name="result">
<guidata><label>Result</label></guidata>
<output id="2" type="string" name="success">
<guidata><label>Success</label></guidata>
</output>
</outputs>
<parameters>
<parameter name="script">
<![CDATA[
function doit()
{
output1 = input1[0];
output2 = "Y";
}
]]>
</parameter>
<parameter name="function">doit</parameter>
</parameters>
</widget>
Match transformation gadgets are added to widgets.xml and are described in exactly the same way as processors (widgets) in EDQ - See Example written processor.
The only limitation is that a match transformation must have only one input and only one output.
Example 2-8 Example script-based gadget for an Upper Case transformation
The following is an example of a script-based gadget for an Upper Case transformation:
The following is an example of a script-based gadget for an Upper Case transformation:
<!-- ************************************************************ -->
<!-- script for a simple uppercase transformation -->
<!-- ************************************************************ -->
<widget id="customUppercase"
class="com.datanomic.director.match.library.util.JavaScriptGadget">
<guidata>
<label>%custom.uppercase.gadget</label>
<group>transformers</group>
<icon>script</icon>
</guidata>
<!-- inputs -->
<inputs>
<input id="1" type="string">
<guidata><label>%customer.uppercase.input</label></guidata>
</input>
</inputs>
<!-- outputs -->
<outputs cardinality="1:1">
<output id="1" type="string" name="result">
<guidata><label>%customer.uppercase.output</label></guidata>
</output>
</outputs>
<parameters>
<parameter name="script">
<![CDATA[
function doit()
{
output1 = input1.toUpperCase();
}
]]>
</parameter>
<parameter name="function">doit</parameter>
</parameters>
</widget>
Using script in written processors
The script is defined as a widget parameter named script. It is entered in an XML CDATA section, to avoid problems with characters like < or > or &.
By default, the entire script is executed for each record processed. If a parameter named function is present, it identifies a function in the script which is called for each record. This is more efficient.
Within the script, each processor input is mapped to the JavaScript name inputN, where N is the input ID from the XML. If the input allows multiple attributes, the value will be an array; otherwise it is a plain value.
Each output from the processor is assigned to the JavaScript name outputN, where N is the output ID from the XML.
First, note that processor options are termed 'properties' internally, and where a property uses Reference Data, this is termed a 'resource' internally.
A packaged script processor can refer to properties and resources.
Each property is mapped to a JavaScript variable with the same name as the property. Non-resource properties are set to the equivalent JavaScript types (String, Number, Date, etc). Each resource property is created as an internal type with methods and properties which allow the resource to be queried.
The following properties and methods are available on resource values:
| Name | Type | Meaning |
|---|---|---|
name |
String | The name of the resource |
keys |
Number | The number of keys (lookup columns) defined for the resource |
results |
Number | The number of result values (return columns) defined for the resource |
types |
Array of Strings | The data types (STRING, NUMBER, DATE) of the keys (lookup columns) and results (return columns) – keys first, then results. |
loadall() |
Array of resource records | Load all the elements of the resource |
query(k1, k2 …) |
Array of resource records | Query the resource – the number and type of the keys must match the resource definition. |
Each resource record returned by loadAll or query is an array containing the key values and result values for the resource record. If there are no matching records, the result of the method call is an empty array.
If the entire contents of a resource are required, a preload parameter should be set on the property definition to indicate that the framework should load the resource once and share the values amongst the process threads, as in:
<resource name="resprop" required="false" structure="1:0" loadable="true">
<guidata><label>Resource property</label></guidata>
<parameters>
<parameter name="preload">true</parameter>
</parameters>
</resource>
The loadAll call will then return the shared array. Given this property definition, the following JavaScript code fragment could be used to output some information on the resource:
var records = resprop.loadall();
output1 = 'Name: ' + resprop.name + "; keys: " + resprop.keys +
"; results: " + resprop.results;
output1 += ' (' + records.length + ' records)'
This is a complete example of a widgets.xml file describing a credit card number validation processor (widget).
The actual details of the credit card validation are omitted, as the objective of this example is to describe how a processor (widget) is constructed.
In this case, the widget has two outputs: a 'success' flag (Y or N) and the credit card type (Amex, Mastercard, etc). The example code uses a mod 10 of the sum of the digits as a lookup into an array of types.
Example 2-9 Complete widgets.xml file
<?xml version="1.0" encoding="UTF-8"?>
<widgets>
<comment>Example packaged JavaScript widgets</comment>
<groupid>1000:custom</groupid>
<!-- ========== -->
<!-- CC checker -->
<!-- ========== -->
<widget id="rde:scdemo" class="com.datanomic.director.widget.scripting.JavaScriptWidget">
<guidata>
<label>Simple CC check demo thing</label>
<group>Custom scripts</group>
</guidata>
<!-- inputs -->
<inputs>
<input id="1" type="string">
<guidata><label>Input CC</label></guidata>
</input>
</inputs>
<!-- outputs -->
<outputs cardinality="1:1">
<output id="1" type="string" name="valid" metadata="true">
<guidata><label>Validity</label></guidata>
</output>
<!-- CC type or message -->
<output id="2" type="string" name="type">
<guidata><label>Type</label></guidata>
</output>
</outputs>
<!-- Spigots -->
<spigots>
<!-- All -->
<spigot id="1">
<guidata><label>All</label></guidata>
</spigot>
<!-- Valid -->
<spigot id="2">
<guidata><label>Valid</label></guidata>
<filter outputid="1" value="Y" equals="true"/>
</spigot>
<!-- Invalid -->
<spigot id="3">
<guidata><label>Invalid</label></guidata>
<filter outputid="1" value="N" equals="true"/>
</spigot>
</spigots>
<!-- Results -->
<results>
<metrics>
<!-- Summary metric: key is valid string -->
<metric id="1" type="fixed">
<fixedrecords fromspigots="true"/>
<drilldowns>
<metricdrill name="types" metricid="2" view="counter"/>
<datadrill name="data" destid="1"/>
</drilldowns>
</metric>
<!-- Frequency count of types -->
<metric id="2" type="variable">
<!-- Simple counter, ignoring no data values -->
<variablerecords keyval="output:2">
<filter outputid="1" value="Y"/>
</variablerecords>
<!-- Drilldown by type -->
<drilldowns>
<datadrill name="bytype" destid="2"/>
</drilldowns>
</metric>
</metrics>
<views default="summary">
<!-- Summary view with:
1. Number and % of records with valid values
2. Number and % of records with invalid values
-->
<view name="summary" viewtype="fixed" metricid="1">
<guidata>
<label>Summary</label>
</guidata>
<columns>
<!-- Number and % of records with valid values-->
<columngroup drilldown="types">
<guidata><label>Valid</label></guidata>
<selector match="Y"/>
<column datum="$count">
<guidata><label>Valid</label></guidata>
</column>
<column datum="$percent" format="0.0">
<guidata><label>%</label></guidata>
</column>
</columngroup>
<!-- Number and % of records with invalid values-->
<columngroup drilldown="data">
<guidata><label>Invalid</label></guidata>
<selector match="N"/>
<column datum="$count">
<guidata><label>Invalid</label></guidata>
</column>
<column datum="$percent" format="0.0">
<guidata><label>%</label></guidata>
</column>
</columngroup>
</columns>
</view>
<!-- Counter view -->
<view name="counter" viewtype="variable" metricid="2" sortby="cp.countcol d" drilldown="bytype">
<guidata>
<label>Credit card types</label>
</guidata>
<!-- Type and count -->
<columns>
<column datum="$key">
<guidata><label>Type</label></guidata>
</column>
<!-- Number and % of records matching the pattern -->
<columnset name="cp" ref="countpercent"/>
</columns>
</view>
</views>
</results>
<!-- Parameters -->
<parameters>
<parameter name="script">
<![CDATA[
///*************************************************************************
// string checkcc([String CardNumber])
//
// Returns CC type or null if invalid. Simple demo mod 10 of sum of digits
// to determine CC type.
//
//*************************************************************************
var types = [ 'Amex', 'Mastercard', 'Visa' ];
function checkcc(ccnumber) {
if (ccnumber == null || ccnumber.length == 0)
return null;
var sum = 0;
for (var i = 0; i < ccnumber.length; i++)
{ var c = ccnumber.charCodeAt(i) - 48;
if (c >= 0 && c <= 9)
sum += c;
else
return null;
}
sum = sum % 10;
return sum > types.length ? null : types[sum];
}
function doit()
{ var type = checkcc(input1);
output1 = type == null ? 'N' : 'Y';
output2 = type == null ? '-' : type;
}
]]>
</parameter>
<parameter name="function">doit</parameter>
</parameters>
</widget>
<!-- Shared columnset for count and percent columns -->
<columnset name="countpercent">
<column name="countcol" datum="$count">
<guidata><label>Count</label></guidata>
</column>
<column name="percentcol" datum="$percent" format="0.0">
<guidata><label>%</label></guidata>
</column>
</columnset>
</widgets>
In some cases, it may be desirable to have separate landing areas for EDQ projects. Project-specific landing areas should be created within the main landing area directory (oedq_local_home/landingarea) and named using the project's internal ID. Using a project's internal ID ensures that a project can be renamed without needing to rename any landing areas or associated scripts.
Creating a project-specific landing area
To create a project-specific landing area:
Determine the internal ID of the project. You can find this information by right-clicking on the project in Director and selecting 'Properties'. The internal ID is on the Advanced tab.
Create a subdirectory in the oedq_local_home/landingarea directory with the same name as the project's internal ID. For the project used in this example, the directory will be called oedq_local_home/landingarea/1
Alternatively, you can use the inbuilt FTP server to create a project-specific landing area. See "Accessing EDQ Files Remotely" in Oracle Fusion Middleware Administering Oracle Enterprise Data Quality for more information.
For information about monitoring real-time processes, see "Monitoring Real-Time Processes" in Oracle Fusion Middleware Administering Oracle Enterprise Data Quality.