Using TMS with Disconnected Systems

The TMS DSI feature provides a customized XML-based batch data load process that enables you to use TMS as a central dictionary classification tool for systems located remotely and operated by different businesses or research organizations.

DSI is designed specifically to allow a pharmaceutical company sponsoring a clinical trial that is being conducted by multiple contract research organizations (CROs), to use TMS as a central classification tool for terminology quality control. In this documentation the central TMS instance is called the sponsor site and the remote classification system instance is called the source site.

CROs send their source data to the sponsor, with contextual information (such as patient ID and document number) and preliminary classifications. The import process at the sponsor site compares the source site's classifications to the sponsor's and loads data in the sponsor database and/or creates warnings or errors associated with faulty data. When the sponsor sends the data back to the source site, the CRO's import process overwrites CRO classifications with those of the sponsor, if there are conflicts, and passes on any Actions created at the sponsor site for particular terms. The import error file shows records that have been overwritten.

The sponsor TMS instance stores all CRO source terms and contextual information, and runs TMS Synchronization on source data (this is automatic during Oracle Clinical Batch Validation).

The sponsor must use TMS. The source site must use either TMS or another dictionary coding system that is capable of processing XML files in the required DSI structure. This section is written for source sites using TMS as their dictionary coding system. For information on the required XML file structure, see Disconnected System Integration XML Files. Disconnected System Integration XML Files also includes the section Validation of Non-TMS XML Files for source sites that do not use TMS.

The external source data system feeding data into TMS can be Oracle Clinical, the Oracle AERS, or a third-party system.

A sponsor can use DSI with any number of source sites. Source sites can use DSI with multiple sponsors, but they must maintain a different database for each sponsor.

The sponsor and source site must synchronize the contents of the verbatim term and classification levels of each dictionary used in a study using DSI.

You can use DSI with databases that support either UTF-8 or Western European (WE) character sets, but both the sponsor and the source site must support the same character set.

Limitations

Disconnected System Integration has the following limitations:

• High-level reclassifications cannot be performed on imported data in the sponsor site. However, high-level reclassifications can be performed at the source site.

• The sponsor instance does not have a UI drill-down capability for source site data.

• If the sponsor is using Oracle Clinical, OC Batch Validation does not run on source site data.

DSI Workflow

The basic DSI workflow follows:

• A source site loads the version of one or more dictionaries specified by the sponsor into TMS or another dictionary coding system.

• The source site may or may not classify its source terms against these dictionaries.

  If the source site is using TMS, it processes terms from the source data system as usual, loading data into TMS and running Autoclassification (with Oracle Clinical as the source data system, Autoclassification runs during Batch Validation). The site can then either send any remaining unclassified terms back to the sponsor as omissions or manually classify them.

• At predefined time intervals, the source site exports newly collected source terms with their contextual information (such as patient ID and document number) and classifications to an XML file with a predefined structure, and sends the file to the sponsor company. If there are source term deletions, they are also included in the file.

• The sponsor site imports the file, compares source site VTAs to its own VTAs, and runs TMS Autoclassification on the omissions in the XML file. If there are any conflicting classifications or other errors, the import process produces error file warnings or errors associated with the appropriate records. The sponsor sets reference codelist values to determine how the import process handles particular problems.

• Sponsor personnel use TMS to manually classify CRO source terms as necessary; creating and approving VTAs, making reclassifications and requesting Actions, according to sponsor standards.

• The sponsor extracts an XML file containing updated classifications, omissions, and Actions associated with the original source terms and their contextual information, and sends it to the source site.

• The source site synchronizes classifications and Actions with the data from the sponsor. If the source site uses TMS, the DSI import process automatically overwrites source site classifications (VTAs) with those of the sponsor's when they conflict.

• The source site sends derived TMS information to its source data system, associated with the appropriate source terms. If the source site uses Oracle Clinical, new Actions and derivations appear in Oracle Clinical after the next Batch Validation.

Both the source site and the sponsor site maintain a full audit trail of each export and import. DSI supports both full and incremental data processing.

For more information, see:

• Setting Up Disconnected System Integration
  
• Using DSI from the Source Site
  
• Using DSI from the Sponsor Site
  
• Viewing XML File and Record Statuses
  
• Viewing Error Files
  
• DSI Batch Jobs
  

Setting Up Disconnected System Integration

To use DSI, you must set it up at both the sponsor and the source sites. Some setup tasks are the same at both sites; others are different. In addition, you must manually ensure that both sites are using the same version of each dictionary for each study.

This section includes:

• Setting Up the Sponsor Instance
  
• Setting Up the Source Instance
  
• Creating Directories on the Database Server
  
• Defining the External Source Data System
  
• Running the DSI Initialization Job
  
• Register Remote Databases
  
• Defining Disconnected System Integrations
  
• Defining X Areas
  
• Defining XML Tag Mappings
  
• Setting DSI Preferences with Reference Codelist Settings
  
• Synchronizing Dictionary Contents Between Sites
  
• Granting User Privileges
  

Setting Up the Sponsor Instance

At the sponsor site you must do the following:

• Create operating system file directories for temporary data and error file storage; see Creating Directories on the Database Server.

• Run the Initialize Disconnected System Integration Systems job with DSI Instance Type set to Sponsor; see Running the DSI Initialization Job.

• Register each remote database by running the job; from the Definitions menu, select Jobs, then choose Register Remote DSI Databases; see Register Remote Databases.

• Enter information about the external source data system(s) (such as Oracle Clinical) used by each source site. You need only define any particular external system once; for example, if you have five source sites using Oracle Clinical, you need only define Oracle Clinical once. If you have already defined Oracle Clinical as an external system because your own organization is using it with TMS, you do not need to define it again. See Defining External System Information in TMS.

• Define each DSI for each remote database/external system combination; see Defining Disconnected System Integrations.

• Map XML tag metadata so that the system can interpret the data received from the source site and send the source site data it can interpret; see Defining XML Tag Mappings.

• Set reference codelist values; see Setting DSI Preferences with Reference Codelist Settings.

• Grant privileges to users who will run the DSI import and export processes; see Granting User Privileges.

• Ensure that the source site has the same version of the dictionary or dictionaries that you are using; see Synchronizing Dictionary Contents Between Sites.

• Ensure that any source site not using TMS knows how to validate XML files against the TMS XML schema before sending them to you; see Validation of Non-TMS XML Files.

Setting Up the Source Instance

At the source site you must do the following:

• Create operating system file directories for temporary data and error file storage; see Creating Directories on the Database Server.

• Enter information about the external source data system(s) (such as Oracle Clinical) whose data you are sending to the sponsor via DSI. If you have already set up the external system for integration with TMS, you do not need to define it again. See Defining External System Information in TMS.

• Run the Initialize Disconnected System Integration Systems job with DSI Instance Type set to Source; see Running the DSI Initialization Job.

• Define Disconnected System Integration with the sponsor; see Defining Disconnected System Integrations.

  Note:

  You must define the external source data system, initialize DSIs, and register the sponsor's database before you can do this step.

• Define X Areas to specify discrete data collection and processing units (such as studies); see Defining X Areas.

• Set reference codelist values; see Setting DSI Preferences with Reference Codelist Settings.

• Grant privileges to users who will run the DSI import and export processes; see Granting User Privileges.

• Ensure that the dictionaries you are using are synchronized with the sponsor's dictionaries; see Synchronizing Dictionary Contents Between Sites.

Creating Directories on the Database Server

You must create three operating system directories on the database server to temporarily hold XML files:

• Input directory. The import process operates on files sent from the remote system to the Input directory.

• Output directory. The export process creates one or more XML files in the Output directory.

• Error directory. Both the import and export processes write an error file to the Error directory if they process any data with errors or warnings.

You must grant Delete privileges on the Input directory to the Oracle User Database Process.

Defining the External Source Data System

You must define each external source data system (such as Oracle Clinical) as an external system in TMS. You need to define each external system only once. If you have already defined an external system because you are using it with TMS, you do not need to define it again.

At a sponsor site, if you have multiple source sites using a particular external source data system, enter information about that system only once.

See Defining External System Information in TMS.

Running the DSI Initialization Job

Before you can export or import data you must initialize your site as either a sponsor or source site. To run the job:

1. Select Definition, then Jobs, then Initialize Disconnected System Integrations. The Initialize Disconnected System Integrations job pop-up window appears.
2. For the job-specific parameter DSI Instance Type, select either Source or Sponsor.
3. If necessary, enter the name of your report server; see Running a Job.
4. Submit the job. See Running a Job.

   Note:

   If you used the DSI feature before installing TMS Release 4.6 and have not yet done so, you must declare your TMS instance to be either a sponsor or a source site by running the DSI initialization job. If you need to act as a sponsor for some studies and a source for other studies, you must set up a different database for each role.

   The initialization job will fail if:

   • you try to initialize as a sponsor but any of your DSI definitions includes your own database as the Instance name.

   • you try to initialize as a source but any of your DSI definitions includes a database other than your own as the Instance name.

   If you have this situation, first delete the X Areas associated with the incorrect instance and then delete the DSI definition that includes the incorrect instance.

Register Remote Databases

At the sponsor site only, register the database used by each source site for DSI by running the job; from the Definition menu, select Jobs, then choose Register Remote DSI Databases. You must register remote databases one at a time.

Note:

Source sites can use DSI with multiple sponsors, but must set up a different database for each sponsor.

The job takes the following parameters:

• Instance. Enter the source site database name.

• External System. From the list, choose the external source data system (such as Oracle Clinical) whose data will be sent to the remote system via DSI.

  Note:

  You must define the external source data system before you can register the remote database. See Defining the External Source Data System.

• Report Server Name. Enter the name of the report server that will run the job.

Defining Disconnected System Integrations

From the Definition menu, select Define Disconnected System Integrations, then the Definitions tab.

Note:

Before defining Disconnected System Integrations here, you must define the external source data system, register the remote database used by each partner, and create directories on the database server. See Defining the External Source Data System, Register Remote Databases, and Creating Directories on the Database Server.

Set up Disconnected System Integration by entering information on both sponsor and source sites:

1. Enter information about each external source data system (such as Oracle Clinical) that the site is using:

   • System. From the list of values, choose the correct external system. The list of values includes all systems defined as external systems in TMS.

   • Instance. From the list of values, choose the correct database location of the external system. The list of values includes all registered remote databases.

2. Remote Partner. Enter the name of the organization that owns the remote database, or other information to help you identify the Disconnected System Integration. The system does not use this field.
3. Enter the names, with the full path, of the Input, Output, and Error directories you created on the database server (see Creating Directories on the Database Server). For example:

   /root/app/oracle/admin/DSI_database_name/xmlworkdir/partnerx_database_input

   where xmlworkdir is your DSI directory, and partnerx_database_input is your input directory for XML files from partner X's remote database.

   Note:

   When you enter this information, the system verifies that the locations are correct. However, if you later change the directories on the server, any export or import job that uses the directory will fail.

4. Save.

Defining X Areas

From the Definition menu, select Define Disconnected Integration Systems, then the X Areas tab.

Define X Areas on the source site only. The first time the XML Import process on the sponsor site encounters data from a particular X Area, it automatically creates a corresponding X Area on the sponsor site.

An X Area is a data collection unit. If you are using Oracle Clinical as your external source data system, X Areas correspond to studies, and the X Area ID is the same as the Study ID. DSI limits the data in any one data exchange to data from a single X Area; by defining an X Area for each study, for example, you ensure that data from only one study is loaded and processed at a time.

During the export/import process, each X Area being processed is locked on both systems, so that data from a particular X Area cannot be exported and imported at the same time. TMS uses the same locks that Oracle Clinical Batch Validation uses; therefore DSI export/import cannot occur at the same time as Oracle Clinical Batch Validation.

You can then change the X Area status at the sponsor site as well as the source site (see Step 3 below).

To define an X Area:

1. In the Define Disconnected System Integrations window's Definitions tab, query for the external system and instance combination for which you need to define an X Area.
2. Click the X Areas tab. The X Areas window opens.
3. Insert a row by selecting Record, then Insert.
4. In the X Area column, enter the ID used by the external source data system for the data collection unit. If Oracle Clinical is the external source data system, enter the Study ID.
5. In the X Area Status column, the status of a new X Area is automatically set to Active. You can change the status later, when appropriate. The possible statuses are:

   • Active. X Areas are automatically set to Active when they are first created. If data from an inactive X Area is sent to either system, the X Area status is automatically changed back to Active.

   • Inactive. Set the X Area's status to Inactive when you no longer expect any data changes. If data is received for the X Area, the import job processes the data, sets the X Area status to Active, and gives a warning.

   • Complete. When you no longer need to use DSI for a particular X Area, you can set the X Area's status to Complete. If data is sent to either system for an X Area with a status of Complete, the import job does not process the data and returns an error.

6. Click in the X Area Name field to display the list of values. The system displays the X Areas defined in the sponsor system and database you specified in the Definitions tab.
7. Select an X Area Name.
8. Enter a description of the X Area in the Description column.
9. Save.

Defining XML Tag Mappings

(Sponsor site only.) From the Definition menu, select Define Disconnected Integration Systems, then the Tag Mappings tab.

XML files exported from both sponsor and source sites use the same tags, but different sites may use different tag values for the same entities. For example, for a particular study a sponsor might use the dictionary MedDRA in a domain called ALLCODING, while a source site might used MedDRA in a domain named SPONSOR_NAME for the same study.

Use this window on the sponsor site only to map the values used by each source site to the names used in your system. The sponsor system uses these mappings to interpret data sent by the source site and to send data to the source site with the names used by the source site.

The system translates the instance and X Area names provided in the XML file to IDs.

For information on the required structure, tags, and generated names of DSI XML files, see Disconnected System Integration XML Files.

For each mapping:

1. Click in the first empty row. The fields become enterable.
2. XML Tag. From the list, select the metadata to map. The choices are:

   • Dictionary Short Name

   • Domain Name

3. Remote System Value. Enter the value used at the source site for a dictionary short name or domain name (whichever you selected in the XML Tag field).

   Note:

   If you are mapping the dictionary short name, and either site is using a virtual dictionary, enter the short name of the associated base dictionary, not the virtual dictionary.

   If you are mapping the domain name, use the virtual domain name if there is one.

4. Central TMS Value. From the list, select the value used at the sponsor site for the same dictionary or domain whose source site name is listed in the XML Value column. See Table 5-1 for details.
5. Save.

Table 5-1 Tag Mapping

XML Tag	Remote System Value	Central TMS Value
DICTIONARY_SHORT_NAME	Source site short name for Dictionary X	Sponsor short name for Dictionary X
DOMAIN_NAME	Source site name for Domain Y	Sponsor name for Domain Y

Note:

If the domain name in either system contains a special character such as (& @ or *) you will not be able to use DSI. You will need to change the domain name.

Setting DSI Preferences with Reference Codelist Settings

The local reference codelist TMS_DSI contains parameters whose settings determine important aspects of DSI behavior. The parameters are: OUTSYNCCSP, CREVTA, GLOBALVTA, and XAPPVTA. The parameters and their settings are described below.

To change the settings for TMS_DSI parameters:

1. From the Definition menu, select Local Reference Codelists.
2. In the Name field, query for TMS_DSI.
3. For each parameter, enter a single-letter setting in the Long Value field and select the Active box. (Selecting Default has no effect.)
4. Save.

OUTSYNCCSP

The DSI import process checks dictionary terms required for new VTAs created at the exporting site against dictionary terms on the verbatim term or classification level at the importing site. If a dictionary term used in a new VTA is not included at the importing site, and the dictionary versions were properly synchronized when DSI was set up, any such difference shows that a dictionary term has been added at the exporting site. The value of the OUTSYNCCSP setting determines the way the import process handles out-of-sync dictionary data.

• Fatal Error (E). If the import process finds a new VTA whose dictionary term is not present at the importing site:

  • The process ends with a Fatal Error status.

  • Information about the missing dictionary term is included in the error file.

  • Any records that were successfully processed (before the record with an error) remain in the database, but also appear in the error file.

• Warning (W). If the import process finds a new VTA whose dictionary term is not present in the importing system:

  • The import job imports source data records as omissions.

  • In the error file, it associates a warning with the VTA record whose dictionary term is missing.

  • On the sponsor site only, the import job creates a Discrepancy Message associated with an omission for the source term whose VTA's dictionary term is missing in the importing system. This Action is visible in the importing system's Classify VT Omissions window. It is not sent to the external source data system, as other Actions are. The Action remains associated with the omission until it is resolved.

• Neither (N). If the import process finds a new VTA whose dictionary term is not present in the importing system:

  • The import job imports source data records as omissions.

  • It creates a Discrepancy Message associated with an omission for the source term whose VTA's dictionary term is missing in the importing system. This Action is visible in the importing system's Classify VT Omissions window. It is not sent to the external source data system, as other Actions are. The Action remains associated with the Omission until it is resolved.

  Note:

  If you choose the setting N, the system does not include these dictionary synchronization errors in the error file. They are visible only in the importing system's Classify Omissions window, where you can query for them in the Action Text field with the string %out of sync%.

The default value (and recommended setting) is E.

CREVTA

(Sponsor site only.) Determines whether the sponsor site imports new VTAs from the source site approved, unapproved, or both. DSI does not use the value entered for this parameter at the source site.

• Approved (A). All VTAs created at the source site are imported as Approved VTAs, regardless of whether they were approved at the source site or not. This setting is not recommended.

• Unapproved (U). All VTAs created at the source site are imported as Unapproved VTAs, regardless of whether they were approved at the source site or not.

• XML (X). VTAs created at the source site are imported with the same approval status—Approved or Unapproved—given by the source site, as reflected in the XML file.

The default value is X.

GLOBALVTA

Determines whether Global VTAs are accepted by the importing system.

• Allow Global VTA (A). Global and Domain VTAs are imported without warnings, if they have no errors.

• Fail Global VTA (F). Global VTAs are not accepted for import. The system creates an error for the record.

• Domain Only (D). Global VTAs are not accepted for import. The system creates a warning for the record and a Domain VTA instead of a Global one.

The default value is A.

XAPPVTA

Enforces whether or not export files must contain only Approved VTAs or can contain Unapproved VTAs as well.

• Yes (Y). Export files must contain only Approved VTAs. Unapproved VTAs are not included in the file. Only when a VTA has been approved is it exported.

• No (N). Export files can contain Approved and Unapproved VTAs.

The default value is N.

Synchronizing Dictionary Contents Between Sites

The source site must use the same version of each dictionary that the sponsor uses for the same study, and dictionary data (including VTA subtypes Accepted and Misspelled) at the two sites must be identical at the verbatim term and classification levels.

When you set up DSI, you must ensure that the sponsor and source site use the same dictionary data; for example, the sponsor can provide dictionary data to load onto the source site. If dictionary synchronization errors do occur, you must correct them.

The DSI import process checks that the coding levels—the verbatim term and classification levels—on the importing site contain all the terms used on the other site. The import process does not compare all terms on the dictionary coding levels of the two dictionaries; it checks only those dictionary terms used in VTAs being processed. You can use the OUTSYNCCSP reference codelist setting to specify what you want the import process to do if the importing site is missing dictionary terms used by the other site. See OUTSYNCCSP.

Note:

If the two systems are using virtual dictionaries, the cut-off dates will probably be different. For example, if the sponsor is using a virtual dictionary and then loads its virtual dictionary onto a source site, the cut-off date on the source site will be later.

Granting User Privileges

You must grant the role TMS_DSI_PRIV to all users who should be able to see the Maintain DSI Files window and run the incremental export and import jobs, all of which are located in the VTA Maintenance menu path, under Disconnected System Integration.

You must grant the role TMS_DEFINE_PRIV to all users who should be able to see the Define Disconnected System Integrations window, run the Force Rederivation job from that window, or to run the Register Remote Databases job. This is the same role that allows access to all the other tasks in the Definition menu path.

For information on granting user privileges, see Assigning Roles to a User.

Using DSI from the Source Site

This section includes:

• Processing Source Data
  
• Exporting Data
  
• Sending the XML File to the Sponsor Site
  
• Importing Data
  
• Updating External System Data
  

Processing Source Data

At the source site, you process terms from the source data system as usual in TMS, loading data into TMS and running Autoclassification. (If you are using Oracle Clinical, this happens automatically during Batch Validation.) The sponsor may or may not want you to manually classify omissions and assign Actions.

Exporting Data

At the source site, you use a batch job to extract source terms and any source term deletions and classifications (VTAs) from your dictionary coding system to an XML file in a predefined format (see Disconnected System Integration XML Files for more information).

On a source site, the export job extracts the following types of data:

• All verbatim term omissions (VTOs) with a timestamp greater than the last export's timestamp

• All source terms with a timestamp greater than the last export's timestamp

• For each source term, contextual information (such as patient ID and document number) and any classifications (VTAs) made at the source site

• New and changed Informative Notes of type Workflow associated with verbatim terms

• Source data deletions, if any

• All VTAs created locally for source data from the selected X Area since the last export of the same X Area

  Note:

  If the local reference codelist TMS_DSI parameter XAPPVT is set to Y, the system exports only Approved VTAs. Nonapproved VTAs are not exported. If set to N, both Approved and Nonapproved VTAs are exported.

Sending the XML File to the Sponsor Site

DSI does not handle transporting the XML file to the sponsor site. You must do it manually; for example, zipping the file and sending it via either FTP or email.

To avoid sending the same files to the sponsor for unnecessary reprocessing, manually delete files from the output directory after sending them to the other site.

Importing Data

After the sponsor processes your data—using Autoclassification and any necessary manual classifications, reclassifications, declassifications, or Action assignments—the sponsor sends an XML file with the updated information to your DSI input directory (see Import DSI Data).

You must run the import job; from the VTA Maintenance menu, select Disconnected System Integration, then choose Import DSI Data.

You can see the status of exports and imports; from the VTA Maintenance menu, select Disconnected System Integration, then Maintain DSI Files. For instructions, see Viewing XML File and Record Statuses.

This section describes how the import process handles the following situations:

Classification Conflicts

The import process compares the VTAs in the XML file with those for the same verbatim terms in your database. If a verbatim term is differently classified by the sponsor, the import process automatically overwrites your VTA with the sponsor's VTA. A warning is associated with the record in the error file so that you can see which terms the sponsor has reclassified.

If the sponsor declassifies a VTA you created, the omission appears in your system. Declassifications occur only if you create a manual classification and the sponsor declassifies the verbatim term without reclassifying it.

Global VTA Conflicts

If the sponsor sends Global VTAs in the XML file, but your organization has set the TMS_DSI local reference codelist value GLOBALVTA to not accept Global VTAs, you will receive errors or warnings for those VTA records, depending on the setting of the codelist value, as follows:

• Fail Global VTA (F). If GLOBALVTA is set to F, and the sponsor sends a Global VTA in the XML file, the system does not import the record. The record remains in the error file with an appropriate error.

• Domain Only (D). If GLOBALVTA is set to D, and the sponsor sends a Global VTA in the XML file, the system creates a Domain VTA for each Global VTA sent, and associates a warning with the record in the error file.

If GLOBALVTA is set to Allow Global VTAs (A), the system imports any Global VTAs sent by the sponsor as Global VTAs, and you never receive Global VTA-related errors or warnings. For more information on GLOBALVTA, see Setting DSI Preferences with Reference Codelist Settings

Source Term Actions

The sponsor may have assigned an Action (either a global or a Discrepancy Message) to one or more omissions, indicating that the source term itself must be changed in the external system before it can be processed in TMS. You can see these Actions in the Classify VT Omissions window.

If you are using Oracle Clinical, these Actions are automatically transmitted to Oracle Clinical during the next Batch Validation. If you use a different external source data system, you must ensure that the Actions are sent to that system.

The import job associates a warning with the omission in the error file in the following situations:

• The omission no longer exists in your system (because it has been manually classified at your site or deleted in the source system).

• The omission exists but is currently owned by the external source data system (Oracle Clinical, for example). This would occur if someone assigned an Action to the omission at your site and sent it to the external system.

Missing Dictionary Terms

The import process compares the dictionary terms used in sponsor VTAs in the file to the dictionary terms on your site. If any dictionary terms required by the sponsor are missing on your site, you must add them, either individually (see Modifying Repository Data in the Maintain Repository Data Window) or by reloading the sponsor's version of the dictionary.

Your organization's setting of the TMS_DSI local reference codelist value OUTSYNCCSP determines how your system handles this situation. The options are:

• OUTSYNCCSP=E (Fatal Error). The import job fails at the first such error it finds. The error file provides information on the missing dictionary term. You cannot import the remaining data in the file until you add the required dictionary term to your database. However, records that were successfully processed before the failed record remain changed in the database. The error file contains a record of each successfully processed record.

  After you add the required dictionary term, you can remove the fatal error message from the XML error file, move the file to the import directory, and run the import process again to process the remaining records.

• OUTSYNCCSP=W (Warning). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. (Unlike the behavior on the sponsor site, the system does not create an Action associated with an omission for the source term whose VTA's dictionary term is missing in the importing system.)

  You must add the dictionary term manually to bring your dictionary into synchronization with the sponsor's, or reload the sponsor's version of the dictionary.

• OUTSYNCCSP=N (Neither). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. There is no mention of the missing dictionary term in the error file.

For more information on OUTSYNCCSP, see Setting DSI Preferences with Reference Codelist Settings

Updating External System Data

You must run the necessary job in the external source data system to synchronize its data with the sponsor site's TMS repository.

If the source data system is Oracle Clinical, the Batch Validation job derives TMS values into Oracle Clinical.

Using DSI from the Sponsor Site

This section includes:

• Importing Data
  
• Processing Source Site Data
  
• Exporting Data
  
• Sending the XML File to the Source Site
  
• Removing X Area Data from the Sponsor Site
  

Importing Data

A source site sends data in one or more XML files to your input directory for processing. Each XML file contains data for a single X Area. Typically, an X Area corresponds to a study; see Defining X Areas.

To import the data into your system, run the import job; from the VTA Maintenance menu, select Disconnected System Integration, then Import DSI Data (see Import DSI Data). The import job runs on all XML files in the input directory.

As part of the import process, Autoclassification runs on all omissions in the X Area, including those sent by the source site.

You can see the status of import files and data; from the VTA Maintenance menu, select Disconnected System Integration, then Maintain DSI Files. For instructions, see Viewing XML File and Record Statuses.

This section describes how the import process handles the following situations:

Adding New Source Terms

Any terms that have been updated (for example, through Oracle Clinical Data Entry Update) or added to TMS from the CRO's external source data system since the last source site data export/sponsor import are included in the XML file and imported by the system, associated with their VTAs, omissions, and associated Informative Notes, if any.

Deleting Source Terms

The source site may mark source terms for deletion if they have been deleted in the external source data system. The import process deletes these terms from your database as well.

Classification Conflicts

The import process compares the VTAs in the XML file with those for the same verbatim terms in your database. If a source site VTA conflicts with your VTA for the same verbatim term—the verbatim term is to a different dictionary term, or one partner has given the VTA a subtype of Misspelled and the other a subtype of Accepted—the system does not import the source site VTA. It remains in the error file, associated with a warning, and the system sets the verbatim term's Update box to Y in your database, forcing the record to be included in your next DSI data export XML file. When the source site imports that file, your VTA overwrites the source site's in the source site's database.

Global VTA Conflicts

If the source site sends Global VTAs in the XML file, but your organization has set the TMS_DSI local reference codelist value GLOBALVTA to not accept Global VTAs, you will receive warnings for those VTA records. The system either does not import the record or imports it as a Domain VTA, depending on the setting of the codelist value, as follows:

• Fail Global VTA (F). If GLOBALVTA is set to F, and the source site sends a Global VTA in the XML file, the system does not import the record. The record remains in the error file associated with an appropriate error.

• Domain Only (D). If GLOBALVTA is set to D, and the source site sends a Global VTA in the XML file, the system creates a Domain VTA for each Global VTA sent, and associates a warning with the record in the error file.

In either of the above cases, the system sets the record's Update box to Y in your database, forcing the record to be included in your next DSI Data Export XML file. When the source site imports that file, your VTA overwrites the source site's in the source site's database.

If GLOBALVTA is set to Allow Global VTAs (A), the system imports any Global VTAs sent by the source site as Global VTAs, and you never receive Global VTA-related errors or warnings.

For more information on GLOBALVTA, see: Setting DSI Preferences with Reference Codelist Settings.

Importing Approved and/or Unapproved VTAs

The setting of the TMS_DSI local reference codelist value CREVTA determines whether your system imports VTAs created by the source site as Approved, Unapproved, or some of each, as follows:

• Unapproved (U). If CREVTA is set to U, the system imports all VTAs created by the source site as Unapproved, regardless of whether or not they were created as Approved at the source site. You must examine each one manually and either approve it or reclassify the verbatim term. See Approving a Nonapproved VTA and Reclassifying a Verbatim Term.

  Note:

  Even if you have this reference codelist set to U, the import job imports VTAs that are direct matches to dictionary terms—that is, VTAs that were created during Autoclassification at the source site—as Approved.

• XML (X). If CREVTA is set to X, the system imports each VTA created by the source site with the same approval status given them at the source site. You must manually process only the Unapproved VTAs. This is the default setting.

• Approved (A). If CREVTA is set to A, the system imports all VTAs created by the source site as Approved, regardless of whether or not they were approved at the source site. No manual processing is necessary. This setting is not recommended.

For more information on CREVTA, see: Setting DSI Preferences with Reference Codelist Settings.

Source Site-Created Actions

The source site may have assigned an external Action to one or more verbatim terms, indicating that the source term itself must be changed in the external system before it can be processed in TMS. These Actions are imported in association with an omission. You can see these Actions in the Classify VT Omissions window.

You can modify source site-created Actions only if the Action is currently owned by TMS. This can happen at two points:

• when the source site has not yet sent the Action to the source data system (via Batch Validation, if the external system is Oracle Clinical)

• when the external system sends the Action back to TMS with a comment or question

Missing Dictionary Terms

The import process compares the dictionary terms used in source site VTAs to the dictionary terms on your site. If any dictionary terms required by the source site are missing on your site, it indicates that the source site has added dictionary terms that your organization does not support. The dictionaries at the two sites must be re-synchronized.

Your organization's setting of the TMS_DSI local reference codelist value OUTSYNCCSP determines how your system handles this situation. The options are:

• OUTSYNCCSP=E (Fatal Error). The import job fails at the first such error it finds. The error file provides information on the missing dictionary term. Records that were successfully processed before the failed record remain changed in the database. The error file contains a record of each successfully processed record.

  Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.

• OUTSYNCCSP=W (Warning). The import job succeeds, but VTAs whose dictionary terms are missing in your system are not imported. The system creates an Action associated with an omission for the source term whose VTA's dictionary term is missing in your system. This Action is visible in the Classify VT Omissions window, as well as in the error file. It is not sent to the external source data system, as other Actions are. The Action remains associated with the omission until it is resolved. You can also read the error file to find these VTAs and the dictionary term they require. Records without errors are successfully imported.

  Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.

• OUTSYNCCSP=N (Neither). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. If an error file is created, it contains no mention of the missing dictionary term.

  The system creates an Action associated with an omission for the source term whose VTA's dictionary term is missing in your system. This Action is visible in the Classify VT Omissions window. It is not sent to the external source data system, as other Actions are, or to the source site. The Action remains associated with the omission until it is resolved.

  Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.

For more information on OUTSYNCCSP, see: Setting DSI Preferences with Reference Codelist Settings

Processing Source Site Data

After running Autoclassification, you must manually process the following source site data. When you update a record, the system sets its Update box to Y, causing the next export job to extract the record to send back to the source site.

Omissions

You must manually classify, or assign Actions to, all the source terms still unclassified after Autoclassification. See Classifying Terms Manually.

Unapproved VTAs

You must examine each Unapproved VTA received from the source site and either approve it or reclassify the verbatim term.

Conflicting Classifications

In the error file, the import process associates warnings with all source site classifications that conflict with VTAs in your database. You can view the conflicts in the error file. The system also sets the Update box to Y for the relevant verbatim term in the database, forcing the record to be included in the next export XML file. During the next source site import, your VTA will automatically overwrite the source site's.

If VTAs conflict because the source site used a dictionary term that does not exist in your repository, the job may fail. See Missing Dictionary Terms.

Exporting Data

When you have finished processing data from the source site, you must extract the updated data for the relevant X Area into an XML file and send it back to the source site. You can either export data from a single X Area (see Export DSI Data) or from all active X Areas at once (see Export Active X Areas). Both jobs export only changed data, using both timestamps and the Update setting to determine which records have been updated.

If necessary, you can force the export of all data by using the Force Rederivation job (see Force Rederivation) to set all records' Update boxes to Y. Then you must run either Export DSI Data or Export Active X Areas to export the data.

Data Exported

The data extract jobs automatically extract the following types of data in the required XML format:

• All verbatim term omissions with new Actions

• All VTAs for verbatim terms based on source terms from the receiving source site that have been created or changed since the last export, including classifications, reclassifications, and approvals

  Note:

  If the local reference codelist TMS_DSI parameter XAPPVTA is set to Y, the system exports only Approved VTAs. Unapproved VTAs are not exported.

• New and changed status audit Informative Notes associated with verbatim terms

• All VTAs declassified since the last export of the same X Area

Export Statuses

You can check on the export's status (see Viewing XML File and Record Statuses). Export jobs can have only two statuses: Success and Fatal Error. The export process should fail only if there is a problem with the Output directory.

Sending the XML File to the Source Site

DSI does not handle transporting the XML file to the source site. You must do it manually; for example, zipping the file and sending it via either FTP or email.

To avoid sending the same files to the partner for unnecessary reprocessing, manually delete files from the output directory after sending them to the other site.

Removing X Area Data from the Sponsor Site

Faulty data may be imported into the sponsor site. Therefore, if you have special privileges, you can delete all data belonging to a specific X Area. See Delete DSI X Areas. You can then reexport all X Area data from the source site (see Force Rederivation) and reimport the X Area.

Viewing XML File and Record Statuses

From the Disconnected System Integration menu, select Maintain DSI Files.

In the Maintain DSI Files window you can view all the import and export jobs associated with each X Area, and see how many records in each file were successfully updated, updated with a warning, and failed. In addition, you can assign a manual status to a file to reflect the state of your work on the file, and view the error file for each job.

For more information, see:

• Viewing File and Record Status
  
• Deleting Files
  
• Setting the Filter
  

Viewing File and Record Status

The Maintain DSI Files window has two sections: X Areas and Files. After you execute a query, the system displays all the X Areas that meet the filter and query criteria in the X Areas section. In the Files section, the system displays all the files that meet the criteria for the X Area selected in the X Areas section.

To view the files for a different X Area, click its name in the X Area column. In the Files section, the system displays its files that meet the filter and query criteria. For each file, the system displays the following information:

• File Name. The file name is generated by the system at the time of export or import (see XML File Names in Disconnected System Integration XML Files for information on the naming convention).

• Process Type: Import or Export.

• Process Status. The way the system determines the process status depends in part on the settings of the local reference codelist TMS_DSI. SeeUsing DSI from the Source Site and Using DSI from the Sponsor Site for the implications of the statuses on your site. The possible statuses are:

  • Success. All records were processed successfully, with no warnings or errors. There is no error file.

  • Warnings. All records were processed successfully, but at least one record has an error. You can see warnings associated with each record with an error in the error file. Depending on the OUTSYNCCSP reference codelist setting, there may be a discrepancy message associated with VTA records whose dictionary term is not included in your dictionary.

  • Errors. Some records may have been processed successfully, but at least one record failed import and remains in the error file associated with an error message.

  • Fatal. The job failed due to a problem with the directory or the XML file itself, or the source site and sponsor dictionaries were out of synch and OUTSYNCCSP was set to fail in that situation.

  For more information on OUTSYNCCSP, see: Setting DSI Preferences with Reference Codelist Settings.

• Manual Status. This status is meant to indicate the state of your work on files with warnings or errors, for use in querying. The system does not enforce any behavior related to the manual status. The possible statuses are:

  • New.

  • Pending.

  • Fixed.

• # Source Data Records. The following three columns concern source term records that are new, updated, or deleted. To see warnings and errors, click View File or select Options, then choose View File.

  • OK: The number of source terms successfully processed without warnings.

  • Warning: The number of source terms successfully processed but with warnings.

  • Failed: The number of source terms with errors; these records were not successfully processed.

• # VTA Data Records. This includes new verbatim term assignments (VTAs, or classifications) reclassified and declassified VTAs, and newly approved or unapproved VTAs. The following three columns concern records that are new VTAs:

  • OK: The number of VTAs successfully processed without warnings.

  • Warning: The number of VTAs successfully processed but with warnings. You can see the warnings in the error file.

  • Failed: The number of VTAs with errors; these records were not successfully processed. You can see the errors in the error file.

• Process Start Time. The time the import or export process began.

• Process End Time. The time the import or export process finished.

• Process Elapsed. The total amount of time spent processing the file, in HH:MM:SS format.

• Audit Information. The following three fields contain audit information for the file selected above:

  • Created By

  • Creation Time

  • Modified By. The user ID of the person who last changed the manual status.

  • Modification Time. The timestamp of the last manual status change.

• Filter. The system displays the current filter settings.

Deleting Files

You can delete X Area files from view by selecting Record, then Delete. This does not delete the file itself, only the database information on the file. You can no longer view the file from this window.

Setting the Filter

The filter determines which X Areas and files are displayed in the Maintain DSI Files window. Your default filter settings are set by selecting the Definition menu, then Maintain Settings.

You can reset the filter at any time by clicking the Filter button in the Maintain DSI Files window. Enter values in the filter pop-up box to limit the files displayed in the Maintain DSI Files window. By default all files are displayed. All settings are optional.

• External System. From the list, select the external system where the source data was originally collected.

• Source Data DB. From the list, select the database of the external system where the source data was originally collected.

• X Area and X Area Name. From the list, select the X Area. The list of values in the X Area field includes both the X Area ID and name, and when you select a value, the system populates both fields.

• X Area Status. From the list, choose the current status of the X Area(s) for the file(s) you want to see. The choices are Active, Inactive, and Complete (for an explanation of the statuses, see Defining X Areas).

• Last Export Between. Enter the start and end dates of the period in which jobs you want to see were executed. Use the format specified for your installation in OPA Settings (OPA_SQL_DATE_FORMAT); for example, if you use the format DD-MON-YYYY, enter 15-OCT-2004.

• Manual Status. From the list, select the manually set status of the file(s) you want to see. The choices are: New, Pending, and Fixed.

• Process Type. From the list, select the process that produced the file you want to see; either Export or Import.

• Process Between…and.… Enter the start and end dates of the period in which jobs you want to see were executed. The system returns all files that were even partially executed during that data range. Use the format specified for your installation in OPA Settings (OPA_SQL_DATE_FORMAT); for example, if you use the format DD-MON-YYYY, enter 15-OCT-2004.

• Process Status. Select one or more status. The system will display files resulting from jobs with the statuses you select, and that meet the other criteria.

Click one of the following buttons:

• Clear. Returns values to what they were when you opened the filter.

• Restore. Resets the filter to the default values (from the Definition menu, select Maintain Settings to set).

• OK. Saves the current settings. The filter pop-up window disappears, leaving the Maintain DSI Files window in view. You must execute a query (from the Query menu, select Execute Query or press F8) to display the files that meet the filter criteria. The filter criteria are displayed at the bottom of the Files section of the window. These settings persist the next time you open the Maintain DSI Files window.

• Cancel. Closes the Filter window without saving any changes.

Viewing Error Files

In the Maintain DSI Files window under the VTA Maintenance menu, you can see the results of each DSI export and import job. If a job had errors, you can look at the error file to find the errors and then fix them and rerun the job.

To find a job, do the following:

1. Click Filter to limit the jobs returned by any or all of the following criteria:

   • External system

   • Source database

   • Remote partner

   • X Area

   • X Area status

   • Last export date range

   • Manual status

   • Process type

   • Process date range

   • Process status (Fatal, Error, Warning, and/or Success)

2. Do one of the following:

   • Click OK to implement these Filter settings for your current TMS session and close the Filter window.

   • Click Cancel to revert settings to their values when you opened the Filter window. The Filter window closes.

   • Click Clear to revert the settings to their values when you opened the Filter window. The Filter window remains open.

   • Click Restore to revert Filter window values to the default settings for your TMS user profile. After clicking Restore, clicking Cancel cannot bring back the settings you had when you opened the filter window. The Filter window remains open.

3. Enter a query in the X Areas block at the top of the window. TMS displays the X Areas matching your filter and query criteria.
4. Select the X Area.
5. Enter a query in the Files block. TMS displays the files matching your filter and query criteria for the X Area you selected.
6. Select the file you want to view and click View File. A browser window opens and displays the file.

The file is in XML format. It is the same XML file that was imported or exported, except that a warning or error is displayed with each record that has a warning or error.

DSI Batch Jobs

Note:

Only superusers can run DSI batch jobs. For information on setting up a superuser account, see Defining a New User and also Running the DSI Initialization Job.

This section includes information on DSI batch jobs:

• Export DSI Data
  
• Import DSI Data
  
• Export Active X Areas
  
• Export X Areas by External System/Instance
  
• Import X Areas by External System/Instance
  
• Delete DSI X Areas
  
• Delete Remote DSI Databases
  
• Force Rederivation
  

Export DSI Data

This job extracts data for a single X Area into an XML file located in the Output directory. It processes only data that has been added, deleted, or updated since the last export job was run ("incremental" data processing). The export process should fail only if there is a problem with the receiving output directory. To run the job:

1. Go to Disconnected System Integration, then select Export DSI Data.
2. Enter values for the following parameters. A list of values is available for the first three.

   • External System. The name of the external system where the source data originated.

   • Instance. The database where the source data originated.

   • X Area. The X Area that corresponds to the unit whose data you want to export; typically a study or case.

   • Report Server Name. Enter the name of the report server you want to run the job.

3. Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.

Import DSI Data

The import process operates on all XML files currently in the input directory, and deletes them from that directory upon completion. If there are errors or warnings, or the job fails, the system inserts the errors into the XML file associated with the appropriate records, and moves the XML file to the error directory.

The import process begins by calling the XML parser to validate the structure of the XML file against the XML schema. If it is invalid, the job fails. The import job also validates the XML metadata against the mappings defined in the sponsor.

The data import job operates on data differently at the source site and the sponsor site; see Using DSI from the Source Site, and Using DSI from the Sponsor Site. The way it handles different errors depends on the settings of several reference codelist values in the importing system; see Setting DSI Preferences with Reference Codelist Settings.

To run the job:

1. Go to Disconnected System Integration, then select Import DSI Data.
2. Enter the name of the report server you want to run the job.
3. Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.

Import Process Statuses

Import jobs can end with four possible statuses: Success, Warning, Error, and Fatal Error. Jobs that end with a status of Success do not have error files. Jobs ending with all other statuses do have error files.

To create the error file, the import process inserts errors or warnings, associated with the appropriate record, into the XML file in the input directory and then moves the XML file into the error directory.

• Success: If all records are imported without errors or warnings, DSI gives the job a status of Success, updates the timestamp of each imported record in the importing database, and deletes the XML file from the input directory. It does not produce an error file.

• Warning: If the import process encounters data errors or conflicts, and you have set up DSI to continue even with errors (see Setting DSI Preferences with Reference Codelist Settings), the import job:

  • successfully imports all records that do not have warnings, and removes those records from the XML file

  • for records with errors or conflicts, does one of the following, depending on reference codelist settings:

    • does not import the record; leaves it in the error file associated with the appropriate error

    • imports the record but associates a warning with the record in the error file; in addition, may associate a Discrepancy Message with the record

• Error: If the import process encounters data errors or conflicts, and you have set up DSI to fail in that situation (see Setting DSI Preferences with Reference Codelist Settings), the import job:

  • does not import records with errors, but keeps them in the error file associated with an error message

  • imports records successfully before encountering the error but also leaves these imported records, and the unprocessed ones listed after the record that caused the error, in the file

• Fatal Error: The following types of errors are fatal and always result in job failure:

  • An input directory does not exist in the importing system, or an output directory does not exist on the exporting system.

  • (Import only) The XML file structure is invalid.

  • (Import only) The dictionary required in the XML file does not exist in the importing system.

  • (Import only) The Oracle database user has not been given Delete privileges for the input directory.

  • (Import only) A new VTA is created using a dictionary term that does not exist in the importing system.

  • (Import only) The input XML file contains the wrong external system name or instance name for the specified X Area.

  The error file includes information about the fatal error and all the records originally contained in the file.

Export Active X Areas

This job extracts data for all active X Areas, creating a separate XML file in the Output directory for each active X Area. It processes only data that has been added, deleted, or updated since the last export job was run ("incremental" data processing). The export process should fail only if there is a problem with the receiving output directory.

1. Go to Disconnected System Integration, then select Export DSI Data.
2. Enter the name of the report server you want to run the job.
3. Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.

Export X Areas by External System/Instance

This job allows you to export data for a particular X Area from a single site. For example, a sponsor may want to export data from a single CRO rather than all CRO participating in the same study (X Area) at the same time.

To run the job:

1. Go to Disconnected System Integration, then select Export X Areas by External System/Instance.
2. From the External System drop-down list, select the external system to which you want to export data.
3. From the Instance drop-down list, select the database to which you want to export data.
4. Enter the name of the report server you want to run the job.
5. Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.

Import X Areas by External System/Instance

This job allows you to import data for a particular X Area from a single site. For example, a sponsor may want to import data from a single CRO rather than all CRO participating in the same study (X Area) at the same time.

To run the job:

1. Go to Disconnected System Integration, then select Import X Areas by External System/Instance.
2. From the External System drop-down list, select the external system from which you want to import data.
3. From the Instance drop-down list, select the database from which you want to import data.
4. Enter the name of the report server you want to run the job.
5. Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Running a Job.

Delete DSI X Areas

Available on the sponsor site only. If faulty data has been loaded into the sponsor site or exported from the source site, it may be necessary to remove all data belonging to the X Area and start again. This job deletes all data belonging to a single X Area: the X Area definition, source terms, omissions, and the database records of the X Area files (not the files themselves).

To run the job:

1. Go to VTA Maintenance, select Disconnected System Integration, then Delete X Area.

2. Enter the following parameters:

   • External System. The name of the external system where the source data originated.

   • Instance. The database where the source data originated.

   • X Area. The X Area that corresponds to the unit whose data you want to export; typically a study or case.

   • Report Server Name. Enter the name of the report server you want to run the job.

If data has become corrupted on the sponsor site, you can export all relevant data from a particular X Area to the other system.

1. Delete the X Area on the sponsor site by running the Delete DSI X Areas job on the sponsor.
2. On the source site, run Force Rederivation on the same X Area.
3. On the source site, run Export DSI Data on the same X Area.
4. Send the data from the source site to the sponsor.
5. Run Import DSI Data on the sponsor site to recreate the X Area and all its data.

Delete Remote DSI Databases

To delete a registered remote database from a sponsor site, do the following:

1. Manually delete all DSI definitions to the remote database in the Define DSIs window, using Delete from the Record menu. This has the effect of deleting all the associated X Areas as well.

2. Run the Delete Remote DSI Databases job.

To run the job:

1. In the Definitions menu, click Jobs and then click Delete Remote DSI Databases. The job window opens.
2. In the job-specific parameter Remote Instance field, select the name of the remote instance from the drop-down list. The system displays only remote instances that are not part of a DSI definition.
3. Run the job. See Running a Job.

Force Rederivation

This job marks all source terms and omissions in a single X Area for processing in the next export process, whether or not they have been added or modified since the last export job ("full" data processing). To actually process the data, you must run the Export DSI Data job.

If data has become corrupted on the other site, or dictionaries are no longer synchronized, you can export all relevant data from a particular X Area to the other system.

1. From the Definition menu, select Define Disconnected System Integrations, then X Areas, and click the Force Rederivation button. The system marks for mandatory processing (Update=Y) all data in this X Area of the types normally exported from the system (which vary, depending on whether the exporting system is a sponsor or a source site).
2. From the DSI Maintenance menu, select Export DSI Data and run the job; see Export DSI Data.