12 Cleaning Data

A discrepancy is associated with a data point identified as incorrect or possibly incorrect. A discrepancy may originate as an Oracle Health Sciences InForm query or be identified in Oracle Health Sciences Data Management Workbench (Oracle DMW) by a validation (edit) check or manual query; see "Reviewing Data".

Use the Discrepancies page to review discrepancies, add comments, apply an action to change their state and/or apply a tag to route them to other reviewers or to InForm or to a spreadsheet to send to the lab where lab data originated.

Your company can set up custom actions, states, categories, and tags to effectively create a custom workflow for discrepancy management. See "Configuring Your Discrepancy Workflow" for information.

Note:

To review all data and raise discrepancies manually, go to the Listings page.

To view data discrepancies in a study:

1. Select the study and a lifecycle mode on the Home page.

2. Select Discrepancies.

Managing Discrepancies

In the Discrepancy Management pane you can review discrepancies, filter the display, and take action on one or more discrepancies.

A discrepancy is associated with its data point in all models, through both upstream and downstream transformations, but the system recognizes it as a single discrepancy. The table and column displayed for each discrepancy are those of the data point in the model in which the discrepancy was created. You can see the model name in the Details pane.

Acting on Multiple Discrepancies

You can select one or more discrepancies and do the same thing to all of them if the action is valid for all the selected discrepancies. You can use Ctrl+Click or Shift+Click to select multiple discrepancies and:

• Actions: Select the action you want to apply from the drop-down list. You must then select and/or enter a reason for change. If the discrepancy is sent to InForm, the reason is visible there.

  Where Is the Action I Want to Apply?

• Add Comment: Enter information that may be helpful in the review process and click OK.

  Internal Comment? is used only for discrepancies that originated in Oracle Thesaurus Management System (TMS). If it is not selected and you add a comment to a discrepancy that was raised in TMS, the system changes the discrepancy's status to TMS EVALUATION and TMS autoclassification runs on the source term during the next transformation job.

• Export All to Excel: When you click the Export All to Excel the system generates an .xls file of all discrepancies that meet the current filter conditions (if any) and puts it in the location you specify.

Modifying a Single Discrepancy

1. Select one discrepancy and click the Edit icon to change:

   • Discrepancy: Edit the text describing the problem with the data or action required.

   • Category: Select a category to apply to the discrepancy. You can filter discrepancies by category.

   • Action: Select an appropriate action to apply to the discrepancy. You must then select and/or enter a reason for change. If the discrepancy is sent to InForm, the reason is visible there.

     Where Is the Action I Want to Apply?

   • Allow Auto Close: If checked, validation checks with the required code can close this discrepancy. If unchecked, this discrepancy must be closed manually even if a validation check that would close it is executed.

   • Reason for Action: If you apply an action to the discrepancy, select and/or enter a reason for the change.

2. Click OK.

Where Is the Action I Want to Apply?

If you selected multiple discrepancies, the system displays only actions that apply a valid next state for all the selected discrepancies. If you select discrepancies that do not share a valid action, the system lists no actions. You must change your selection to a set of discrepancies that are in the same state. They may also need to have the same tag applied, depending on how your company uses the system; see "Creating a Custom Workflow by Creating and Editing Actions".

If you selected a single discrepancy and the discrepant data point is derived from multiple data values from different sources, the system uses the data source identified as Preferred in the transformation to determine which routing action to enable. If no action is available to send the discrepancy to the source you need, you may need to enter the discrepancy against the data point in the source data model.

Discrepancy Display

Most of the columns displayed across the Discrepancy Management pane are the same as those listed under Detail when a single discrepancy is selected; see "Viewing Discrepancy Details" for descriptions.

You can reorder columns in the page by dragging them.

Click the Refresh icon to show any changes that have occurred since you opened the window. Other users may be working on the discrepancies you are viewing.

The icon displayed indicates the current state of the discrepancy. The state is also displayed as text in the State column and in the Detail pane.

Table 12-1 Discrepancy Icons

	Candidate—Requires manual review before opening.
	Open—Requires resolution.
	Answered—Additional information or a data change is available. Requires manual review and resolution.
	Closed—No further action is allowed.
	Cancelled—No further action is allowed
	Sent to Spreadsheet or Sent to InForm—Normally for the purpose of sending to the source lab. No further action is allowed.

Managing TMS Discrepancies

If your study is set up to use Oracle Thesaurus Management System (TMS) for coding source data to standard terminologies, or dictionaries, TMS processes designated items that have new or changed data during transformations. If TMS is able to automatically code a designated source term to a dictionary term, it does so and derives specified data back to Oracle DMW.

TMS creates a discrepancy when it cannot automatically code a designated source term to a dictionary term. A TMS user must examine the data and either:

• Classify it manually—possible only in the production lifecycle. TMS then closes the discrepancy and derives data to Oracle DMW.

• Send text to Oracle DMW as discrepancy text describing a problem with the data or an action required. For example, if the item is "headache and nausea" the message could be "Split the term."

You can look at a discrepancy's history to see all past discrepancy texts. If TMS autoclassification was unable to code the term, the discrepancy has standard text, "TMS unable to classify term."

Oracle DMW uses four TMS-related system categories, displayed in the Categories column, to help process these discrepancies:

• TMS in Progress prevents updates in Oracle DMW while the item (term) awaits manual classification in TMS.

• TMS Evaluation means that the item or discrepancy has been updated either in the source system or in Oracle DMW, and triggers processing the term in TMS during the next transformation.

• TMS DM Review means that the data manager should review the discrepancy in Oracle DMW.

• TMS Inv Review sends the discrepancy to InForm as a query. The system also sends any subsequent updates to the discrepancy to InForm. The investigator should review the query.

  You cannot change the status of TMS discrepancies, but the status can be changed in InForm as well as TMS. This may cause conflicts. For example, an InForm user might close a discrepancy thinking the term is a valid one while TMS does not recognize it. In this case, even though the Closed status is loaded into Oracle DMW, TMS opens a new discrepancy when autoclassification fails.

  See "Setting Up Coding in TMS" for more information.

Viewing Data and Queries in InForm

In the Details pane, click the View Data in InForm icon or the View Query in InForm icon. The InForm Login screen opens. After you log in, the system displays the selected query or its underlying data in context.

Note:

The InForm URL must be defined in the InForm configuration for the current data model and lifecycle.

Viewing Discrepancy Details

The Details pane displays all current information about a discrepancy, including the data value. You cannot make changes here.

• Subject: Subject ID.

• Visit: Name of the visit.

• Discrepancy: Discrepancy text. You can edit this in the Discrepancy Management pane by clicking the Edit icon.

• Table: The database table that contains the discrepant value.

• Model: The clinical data model that contains the data value against which the discrepancy was originally created.

  Note:

  If the model is InForm, the discrepancy originated as a query in InForm.

• Item: The table column that contains the discrepant value.

• Value: The discrepant data point value

• State: The state currently applied to the discrepancy: Candidate, Open, Cancelled, Answered, or Closed. You can change the state in the Discrepancy Management pane by selecting a value from the Actions drop-down list.

• Days in State: The number of days the current state has been applied to the discrepancy.

• Category: The category assigned to the discrepancy, if any. You can filter by category. A discrepancy can have only one category at a time. You can change the category in the Discrepancy Management pane by clicking the Edit icon.

• Created: The username of the person who created the discrepancy and the date it was created. If the user created the discrepancy as a query in InForm, the InForm username is displayed and "InForm" is displayed.

• Modified: The username of the person who most recently modified the discrepancy and the date it was modified. If the user modifed the discrepancy as a query in InForm, the InForm username followed by "(InForm)" is displayed.

• Tag: The tag assigned to the discrepancy, if any. Some actions apply tags.

• New Data: If Yes, the data value has been changed in the source system and the discrepancy has not been reviewed in Oracle DMW since then. When you or someone else clicks Display Full Record, the system considers that you have reviewed it and sets this value to No. You can also set it to No.

• View Query in InForm: When you click this icon, the InForm Login screen opens. After you log in, the system displays the selected query in context.

• View Data in InForm: When you click this icon, the InForm Login screen opens. After you log in, the system displays the underlying data in context.

Viewing Discrepancy History

The History pane displays the history of the discrepancy—by default in reverse chronological order, with the most recent change at the top. The user who made the change and the date of the change are displayed. You cannot make changes to the data but you can change the display; see "Changing the User Interface Display".

Select Refresh to view any changes that have occurred since you selected the discrepancy or last refreshed.

Displaying the Full Record

The Display Full Record pane displays all column values for the record containing the discrepant data point, including the current value and the original value, if it is different. The discrepant item is always listed at the top.

Two internal columns are displayed at the bottom:

• CDR$SKEY contains the primary key values of the record separated by tildes (~). This value is used internally to trace data lineage; see "How the System Tracks Data Lineage".

• CDR$DUP_NUM is normally null, but can contain an integer that indicates the record's place relative to other records loaded from the same source with the same primary key; see "Supporting Duplicate Primary Keys in a Load".

Select Refresh to view any changes that have occurred since you selected the discrepancy or last refreshed.

Creating and Using Discrepancy Filters

Use filters to show only discrepancies that meet certain criteria. You can name and save filters to use later, use filters other people have created and made public or shared, copy existing filters, group filters, or just use a filter during your current session; see "Creating and Using Filters".

The system continues to apply filters set in the Listings pages in the Discrepancies page, except:

• Data Changed filters are ignored.

• A Subject or Visit filter set to apply only to a specific data model is interpreted as an additional filter restricting discrepancies to just those in that data model.

• If different Subject and Visit filters specify different data models as the driving data model and they are applied at the same time, the system ignores all Subject and Visit filters.

A message appears when any of these situations occur.

Discrepancy filters include:

• "Creating and Editing Location (Tables and Models) Filters"

• "Creating and Editing Data Source Filters"

• "Creating and Editing Discrepancy States and Tags Filters"

• "Creating and Editing Discrepancy User Filters"

• "Creating and Editing Subject Visit Flags Filters"

• "Creating and Editing Record Flags Filters"

• "Creating and Editing Discrepancy Origin Filters"

Creating and Editing Data Source Filters

Use this filter to retrieve discrepancies by their original data source or data source type—for example, a lab or InForm.

1. Filter Type. Select the way you want to search:

   • Datasource The system lists InForm and all labs used in the current study.

   • Datasource Type The system lists Labs (for all labs used in the current study), InForm, and Unknown.

2. Select the data source or type to search. You can enter part or all of a model or table name above the column to filter.

3. Check Exclude Based On Criteria to filter out records that satisfy the criteria. This is the equivalent of adding "Not" to the query logic. Leave unchecked to display only the records that satisfy the criteria.

4. Filter Availability is All Studies. Saved filters of this type are available for use in all studies in the current lifecycle.

5. Filter Name: Enter a descriptive name for the filter to save it to use in another session; that is, after you log out or work in a different study.

   Filter names must be unique across all private, shared, and public filters that you can see in a particular study and lifecycle.

   If you do not name the filter, the system calls it Temporary Filter and it is available only during your current session.

6. Description: Enter additional text to help you and other users know if this filter meets their needs. The description is displayed in the Edit Filter window and in the Current Filters window that lists all filters currently applied on a page.

7. Save As Copy. This option is available only when you are editing an existing, named filter. If this option is selected, you can make changes and save the filter with a different name.

8. Click OK

Creating and Editing Location (Tables and Models) Filters

Use this filter to retrieve discrepancies by the table or data model against which they were originally raised.

1. Filter Type. Select the structure you want to search:

   • Model The system lists the models in the current study.

   • Table The system lists the models and tables in the current study.

2. Select one or more models or tables. You can enter part or all of a model or table name above the column to filter.

3. Check Exclude Based On Criteria to filter out records that satisfy the criteria. This is the equivalent of adding "Not" to the query logic. Leave unchecked to display only the records that satisfy the criteria.

4. Filter Availability is Study. Saved filters of this type are available for use only in the current study.

5. Filter Name: Enter a descriptive name for the filter to save it to use in another session; that is, after you log out or work in a different study.

   Filter names must be unique across all private, shared, and public filters that you can see in a particular study and lifecycle.

   If you do not name the filter, the system calls it Temporary Filter and it is available only during your current session.

6. Description: Enter additional text to help you and other users know if this filter meets their needs. The description is displayed in the Edit Filter window and in the Current Filters window that lists all filters currently applied on a page.

7. Save As Copy. This option is available only when you are editing an existing, named filter. If this option is selected, you can make changes and save the filter with a different name.

8. Click OK

Exporting All to Excel

Click Export to Excel to generate an .xls file that includes all discrepancies that satisfy the current filters (if any).

Adding a Comment

Select one or more discrepancies and click the Add Comment icon to add the same comment to all selected discrepancies.

The system uses the Internal Comment? check box only for discrepancies that originated in Oracle Thesaurus Management System (TMS). If it is not selected and you add a comment to a discrepancy that was raised in TMS, the system changes the discrepancy's status to TMS EVALUATION and TMS autoclassification runs on the source term during the next transformation job.

Actions

The Actions drop-down lists all valid actions for the selected discrepancies. To apply an action, select it from the list. A window appears.

1. In the window, either:

   • Select a standard reason for change.

   • Enter your own reason.

   • Select a reason and enter additional information.

2. Click OK.