Using TMS with Disconnected Systems
The TMS DSI feature provides a customized XML-based batch data load process that enables you to use TMS as a central dictionary classification tool for systems located remotely and operated by different businesses or research organizations.
DSI is designed specifically to allow a pharmaceutical company sponsoring a clinical trial that is being conducted by multiple contract research organizations (CROs), to use TMS as a central classification tool for terminology quality control. In this documentation the central TMS instance is called the sponsor site and the remote classification system instance is called the source site.
CROs send their source data to the sponsor, with contextual information (such as patient ID and document number) and preliminary classifications. The import process at the sponsor site compares the source site's classifications to the sponsor's and loads data in the sponsor database and/or creates warnings or errors associated with faulty data. When the sponsor sends the data back to the source site, the CRO's import process overwrites CRO classifications with those of the sponsor, if there are conflicts, and passes on any Actions created at the sponsor site for particular terms. The import error file shows records that have been overwritten.
The sponsor TMS instance stores all CRO source terms and contextual information, and runs TMS Synchronization on source data (this is automatic during Oracle Clinical Batch Validation).
The sponsor must use TMS. The source site must use either TMS or another dictionary coding system that is capable of processing XML files in the required DSI structure. This section is written for source sites using TMS as their dictionary coding system. For information on the required XML file structure, see Disconnected System Integration XML Files. Disconnected System Integration XML Files also includes the section Validation of Non-TMS XML Files for source sites that do not use TMS.
The external source data system feeding data into TMS can be Oracle Clinical, the Oracle AERS, or a third-party system.
A sponsor can use DSI with any number of source sites. Source sites can use DSI with multiple sponsors, but they must maintain a different database for each sponsor.
The sponsor and source site must synchronize the contents of the verbatim term and classification levels of each dictionary used in a study using DSI.
You can use DSI with databases that support either UTF-8 or Western European (WE) character sets, but both the sponsor and the source site must support the same character set.
Limitations
Disconnected System Integration has the following limitations:
-
High-level reclassifications cannot be performed on imported data in the sponsor site. However, high-level reclassifications can be performed at the source site.
-
The sponsor instance does not have a UI drill-down capability for source site data.
-
If the sponsor is using Oracle Clinical, OC Batch Validation does not run on source site data.
DSI Workflow
The basic DSI workflow follows:
-
A source site loads the version of one or more dictionaries specified by the sponsor into TMS or another dictionary coding system.
-
The source site may or may not classify its source terms against these dictionaries.
If the source site is using TMS, it processes terms from the source data system as usual, loading data into TMS and running Autoclassification (with Oracle Clinical as the source data system, Autoclassification runs during Batch Validation). The site can then either send any remaining unclassified terms back to the sponsor as omissions or manually classify them.
-
At predefined time intervals, the source site exports newly collected source terms with their contextual information (such as patient ID and document number) and classifications to an XML file with a predefined structure, and sends the file to the sponsor company. If there are source term deletions, they are also included in the file.
-
The sponsor site imports the file, compares source site VTAs to its own VTAs, and runs TMS Autoclassification on the omissions in the XML file. If there are any conflicting classifications or other errors, the import process produces error file warnings or errors associated with the appropriate records. The sponsor sets reference codelist values to determine how the import process handles particular problems.
-
Sponsor personnel use TMS to manually classify CRO source terms as necessary; creating and approving VTAs, making reclassifications and requesting Actions, according to sponsor standards.
-
The sponsor extracts an XML file containing updated classifications, omissions, and Actions associated with the original source terms and their contextual information, and sends it to the source site.
-
The source site synchronizes classifications and Actions with the data from the sponsor. If the source site uses TMS, the DSI import process automatically overwrites source site classifications (VTAs) with those of the sponsor's when they conflict.
-
The source site sends derived TMS information to its source data system, associated with the appropriate source terms. If the source site uses Oracle Clinical, new Actions and derivations appear in Oracle Clinical after the next Batch Validation.
Both the source site and the sponsor site maintain a full audit trail of each export and import. DSI supports both full and incremental data processing.
For more information, see:
- Setting Up Disconnected System Integration
- Using DSI from the Source Site
- Using DSI from the Sponsor Site
- Viewing XML File and Record Statuses
- Viewing Error Files
- DSI Batch Jobs
Parent topic: Integrating TMS with Other Systems
Setting Up Disconnected System Integration
To use DSI, you must set it up at both the sponsor and the source sites. Some setup tasks are the same at both sites; others are different. In addition, you must manually ensure that both sites are using the same version of each dictionary for each study.
This section includes:
- Setting Up the Sponsor Instance
- Setting Up the Source Instance
- Creating Directories on the Database Server
- Defining the External Source Data System
- Running the DSI Initialization Job
- Register Remote Databases
- Defining Disconnected System Integrations
- Defining X Areas
- Defining XML Tag Mappings
- Setting DSI Preferences with Reference Codelist Settings
- Synchronizing Dictionary Contents Between Sites
- Granting User Privileges
Parent topic: Using TMS with Disconnected Systems
Setting Up the Sponsor Instance
At the sponsor site you must do the following:
-
Create operating system file directories for temporary data and error file storage; see Creating Directories on the Database Server.
-
Run the Initialize Disconnected System Integration Systems job with DSI Instance Type set to Sponsor; see Running the DSI Initialization Job.
-
Register each remote database by running the job; from the Definitions menu, select Jobs, then choose Register Remote DSI Databases; see Register Remote Databases.
-
Enter information about the external source data system(s) (such as Oracle Clinical) used by each source site. You need only define any particular external system once; for example, if you have five source sites using Oracle Clinical, you need only define Oracle Clinical once. If you have already defined Oracle Clinical as an external system because your own organization is using it with TMS, you do not need to define it again. See Defining External System Information in TMS.
-
Define each DSI for each remote database/external system combination; see Defining Disconnected System Integrations.
-
Map XML tag metadata so that the system can interpret the data received from the source site and send the source site data it can interpret; see Defining XML Tag Mappings.
-
Set reference codelist values; see Setting DSI Preferences with Reference Codelist Settings.
-
Grant privileges to users who will run the DSI import and export processes; see Granting User Privileges.
-
Ensure that the source site has the same version of the dictionary or dictionaries that you are using; see Synchronizing Dictionary Contents Between Sites.
-
Ensure that any source site not using TMS knows how to validate XML files against the TMS XML schema before sending them to you; see Validation of Non-TMS XML Files.
Parent topic: Setting Up Disconnected System Integration
Setting Up the Source Instance
At the source site you must do the following:
-
Create operating system file directories for temporary data and error file storage; see Creating Directories on the Database Server.
-
Enter information about the external source data system(s) (such as Oracle Clinical) whose data you are sending to the sponsor via DSI. If you have already set up the external system for integration with TMS, you do not need to define it again. See Defining External System Information in TMS.
-
Run the Initialize Disconnected System Integration Systems job with DSI Instance Type set to Source; see Running the DSI Initialization Job.
-
Define Disconnected System Integration with the sponsor; see Defining Disconnected System Integrations.
Note:
You must define the external source data system, initialize DSIs, and register the sponsor's database before you can do this step.
-
Define X Areas to specify discrete data collection and processing units (such as studies); see Defining X Areas.
-
Set reference codelist values; see Setting DSI Preferences with Reference Codelist Settings.
-
Grant privileges to users who will run the DSI import and export processes; see Granting User Privileges.
-
Ensure that the dictionaries you are using are synchronized with the sponsor's dictionaries; see Synchronizing Dictionary Contents Between Sites.
Parent topic: Setting Up Disconnected System Integration
Creating Directories on the Database Server
You must create three operating system directories on the database server to temporarily hold XML files:
-
Input directory. The import process operates on files sent from the remote system to the Input directory.
-
Output directory. The export process creates one or more XML files in the Output directory.
-
Error directory. Both the import and export processes write an error file to the Error directory if they process any data with errors or warnings.
You must grant Delete privileges on the Input directory to the Oracle User Database Process.
Parent topic: Setting Up Disconnected System Integration
Defining the External Source Data System
You must define each external source data system (such as Oracle Clinical) as an external system in TMS. You need to define each external system only once. If you have already defined an external system because you are using it with TMS, you do not need to define it again.
At a sponsor site, if you have multiple source sites using a particular external source data system, enter information about that system only once.
Parent topic: Setting Up Disconnected System Integration
Running the DSI Initialization Job
Before you can export or import data you must initialize your site as either a sponsor or source site. To run the job:
Parent topic: Setting Up Disconnected System Integration
Register Remote Databases
At the sponsor site only, register the database used by each source site for DSI by running the job; from the Definition menu, select Jobs, then choose Register Remote DSI Databases. You must register remote databases one at a time.
Note:
Source sites can use DSI with multiple sponsors, but must set up a different database for each sponsor.
The job takes the following parameters:
-
Instance. Enter the source site database name.
-
External System. From the list, choose the external source data system (such as Oracle Clinical) whose data will be sent to the remote system via DSI.
Note:
You must define the external source data system before you can register the remote database. See Defining the External Source Data System.
-
Report Server Name. Enter the name of the report server that will run the job.
Parent topic: Setting Up Disconnected System Integration
Defining Disconnected System Integrations
From the Definition menu, select Define Disconnected System Integrations, then the Definitions tab.
Note:
Before defining Disconnected System Integrations here, you must define the external source data system, register the remote database used by each partner, and create directories on the database server. See Defining the External Source Data System, Register Remote Databases, and Creating Directories on the Database Server.
Set up Disconnected System Integration by entering information on both sponsor and source sites:
Parent topic: Setting Up Disconnected System Integration
Defining X Areas
From the Definition menu, select Define Disconnected Integration Systems, then the X Areas tab.
Define X Areas on the source site only. The first time the XML Import process on the sponsor site encounters data from a particular X Area, it automatically creates a corresponding X Area on the sponsor site.
An X Area is a data collection unit. If you are using Oracle Clinical as your external source data system, X Areas correspond to studies, and the X Area ID is the same as the Study ID. DSI limits the data in any one data exchange to data from a single X Area; by defining an X Area for each study, for example, you ensure that data from only one study is loaded and processed at a time.
During the export/import process, each X Area being processed is locked on both systems, so that data from a particular X Area cannot be exported and imported at the same time. TMS uses the same locks that Oracle Clinical Batch Validation uses; therefore DSI export/import cannot occur at the same time as Oracle Clinical Batch Validation.
You can then change the X Area status at the sponsor site as well as the source site (see Step 3 below).
To define an X Area:
Parent topic: Setting Up Disconnected System Integration
Defining XML Tag Mappings
(Sponsor site only.) From the Definition menu, select Define Disconnected Integration Systems, then the Tag Mappings tab.
XML files exported from both sponsor and source sites use the same tags, but different sites may use different tag values for the same entities. For example, for a particular study a sponsor might use the dictionary MedDRA in a domain called ALLCODING, while a source site might used MedDRA in a domain named SPONSOR_NAME for the same study.
Use this window on the sponsor site only to map the values used by each source site to the names used in your system. The sponsor system uses these mappings to interpret data sent by the source site and to send data to the source site with the names used by the source site.
The system translates the instance and X Area names provided in the XML file to IDs.
For information on the required structure, tags, and generated names of DSI XML files, see Disconnected System Integration XML Files.
For each mapping:
Table 5-1 Tag Mapping
XML Tag | Remote System Value | Central TMS Value |
---|---|---|
DICTIONARY_SHORT_NAME |
Source site short name for Dictionary X |
Sponsor short name for Dictionary X |
DOMAIN_NAME |
Source site name for Domain Y |
Sponsor name for Domain Y |
Note:
If the domain name in either system contains a special character such as (& @ or *) you will not be able to use DSI. You will need to change the domain name.
Parent topic: Setting Up Disconnected System Integration
Setting DSI Preferences with Reference Codelist Settings
The local reference codelist TMS_DSI contains parameters whose settings determine important aspects of DSI behavior. The parameters are: OUTSYNCCSP, CREVTA, GLOBALVTA, and XAPPVTA. The parameters and their settings are described below.
To change the settings for TMS_DSI parameters:
- From the Definition menu, select Local Reference Codelists.
- In the Name field, query for TMS_DSI.
- For each parameter, enter a single-letter setting in the Long Value field and select the Active box. (Selecting Default has no effect.)
- Save.
OUTSYNCCSP
The DSI import process checks dictionary terms required for new VTAs created at the exporting site against dictionary terms on the verbatim term or classification level at the importing site. If a dictionary term used in a new VTA is not included at the importing site, and the dictionary versions were properly synchronized when DSI was set up, any such difference shows that a dictionary term has been added at the exporting site. The value of the OUTSYNCCSP setting determines the way the import process handles out-of-sync dictionary data.
-
Fatal Error (E). If the import process finds a new VTA whose dictionary term is not present at the importing site:
-
The process ends with a Fatal Error status.
-
Information about the missing dictionary term is included in the error file.
-
Any records that were successfully processed (before the record with an error) remain in the database, but also appear in the error file.
-
-
Warning (W). If the import process finds a new VTA whose dictionary term is not present in the importing system:
-
The import job imports source data records as omissions.
-
In the error file, it associates a warning with the VTA record whose dictionary term is missing.
-
On the sponsor site only, the import job creates a Discrepancy Message associated with an omission for the source term whose VTA's dictionary term is missing in the importing system. This Action is visible in the importing system's Classify VT Omissions window. It is not sent to the external source data system, as other Actions are. The Action remains associated with the omission until it is resolved.
-
-
Neither (N). If the import process finds a new VTA whose dictionary term is not present in the importing system:
-
The import job imports source data records as omissions.
-
It creates a Discrepancy Message associated with an omission for the source term whose VTA's dictionary term is missing in the importing system. This Action is visible in the importing system's Classify VT Omissions window. It is not sent to the external source data system, as other Actions are. The Action remains associated with the Omission until it is resolved.
Note:
If you choose the setting N, the system does not include these dictionary synchronization errors in the error file. They are visible only in the importing system's Classify Omissions window, where you can query for them in the Action Text field with the string
%out of sync%
. -
The default value (and recommended setting) is E.
CREVTA
(Sponsor site only.) Determines whether the sponsor site imports new VTAs from the source site approved, unapproved, or both. DSI does not use the value entered for this parameter at the source site.
-
Approved (A). All VTAs created at the source site are imported as Approved VTAs, regardless of whether they were approved at the source site or not. This setting is not recommended.
-
Unapproved (U). All VTAs created at the source site are imported as Unapproved VTAs, regardless of whether they were approved at the source site or not.
-
XML (X). VTAs created at the source site are imported with the same approval status—Approved or Unapproved—given by the source site, as reflected in the XML file.
The default value is X.
GLOBALVTA
Determines whether Global VTAs are accepted by the importing system.
-
Allow Global VTA (A). Global and Domain VTAs are imported without warnings, if they have no errors.
-
Fail Global VTA (F). Global VTAs are not accepted for import. The system creates an error for the record.
-
Domain Only (D). Global VTAs are not accepted for import. The system creates a warning for the record and a Domain VTA instead of a Global one.
The default value is A.
XAPPVTA
Enforces whether or not export files must contain only Approved VTAs or can contain Unapproved VTAs as well.
-
Yes (Y). Export files must contain only Approved VTAs. Unapproved VTAs are not included in the file. Only when a VTA has been approved is it exported.
-
No (N). Export files can contain Approved and Unapproved VTAs.
The default value is N.
Parent topic: Setting Up Disconnected System Integration
Synchronizing Dictionary Contents Between Sites
The source site must use the same version of each dictionary that the sponsor uses for the same study, and dictionary data (including VTA subtypes Accepted and Misspelled) at the two sites must be identical at the verbatim term and classification levels.
When you set up DSI, you must ensure that the sponsor and source site use the same dictionary data; for example, the sponsor can provide dictionary data to load onto the source site. If dictionary synchronization errors do occur, you must correct them.
The DSI import process checks that the coding levels—the verbatim term and classification levels—on the importing site contain all the terms used on the other site. The import process does not compare all terms on the dictionary coding levels of the two dictionaries; it checks only those dictionary terms used in VTAs being processed. You can use the OUTSYNCCSP reference codelist setting to specify what you want the import process to do if the importing site is missing dictionary terms used by the other site. See OUTSYNCCSP.
Note:
If the two systems are using virtual dictionaries, the cut-off dates will probably be different. For example, if the sponsor is using a virtual dictionary and then loads its virtual dictionary onto a source site, the cut-off date on the source site will be later.
Parent topic: Setting Up Disconnected System Integration
Granting User Privileges
You must grant the role TMS_DSI_PRIV to all users who should be able to see the Maintain DSI Files window and run the incremental export and import jobs, all of which are located in the VTA Maintenance menu path, under Disconnected System Integration.
You must grant the role TMS_DEFINE_PRIV to all users who should be able to see the Define Disconnected System Integrations window, run the Force Rederivation job from that window, or to run the Register Remote Databases job. This is the same role that allows access to all the other tasks in the Definition menu path.
For information on granting user privileges, see Assigning Roles to a User.
Parent topic: Setting Up Disconnected System Integration
Using DSI from the Source Site
This section includes:
- Processing Source Data
- Exporting Data
- Sending the XML File to the Sponsor Site
- Importing Data
- Updating External System Data
Parent topic: Using TMS with Disconnected Systems
Processing Source Data
At the source site, you process terms from the source data system as usual in TMS, loading data into TMS and running Autoclassification. (If you are using Oracle Clinical, this happens automatically during Batch Validation.) The sponsor may or may not want you to manually classify omissions and assign Actions.
Parent topic: Using DSI from the Source Site
Exporting Data
At the source site, you use a batch job to extract source terms and any source term deletions and classifications (VTAs) from your dictionary coding system to an XML file in a predefined format (see Disconnected System Integration XML Files for more information).
On a source site, the export job extracts the following types of data:
-
All verbatim term omissions (VTOs) with a timestamp greater than the last export's timestamp
-
All source terms with a timestamp greater than the last export's timestamp
-
For each source term, contextual information (such as patient ID and document number) and any classifications (VTAs) made at the source site
-
New and changed Informative Notes of type Workflow associated with verbatim terms
-
Source data deletions, if any
-
All VTAs created locally for source data from the selected X Area since the last export of the same X Area
Note:
If the local reference codelist TMS_DSI parameter XAPPVT is set to Y, the system exports only Approved VTAs. Nonapproved VTAs are not exported. If set to N, both Approved and Nonapproved VTAs are exported.
Parent topic: Using DSI from the Source Site
Sending the XML File to the Sponsor Site
DSI does not handle transporting the XML file to the sponsor site. You must do it manually; for example, zipping the file and sending it via either FTP or email.
To avoid sending the same files to the sponsor for unnecessary reprocessing, manually delete files from the output directory after sending them to the other site.
Parent topic: Using DSI from the Source Site
Importing Data
After the sponsor processes your data—using Autoclassification and any necessary manual classifications, reclassifications, declassifications, or Action assignments—the sponsor sends an XML file with the updated information to your DSI input directory (see Import DSI Data).
You must run the import job; from the VTA Maintenance menu, select Disconnected System Integration, then choose Import DSI Data.
You can see the status of exports and imports; from the VTA Maintenance menu, select Disconnected System Integration, then Maintain DSI Files. For instructions, see Viewing XML File and Record Statuses.
This section describes how the import process handles the following situations:
Classification Conflicts
The import process compares the VTAs in the XML file with those for the same verbatim terms in your database. If a verbatim term is differently classified by the sponsor, the import process automatically overwrites your VTA with the sponsor's VTA. A warning is associated with the record in the error file so that you can see which terms the sponsor has reclassified.
If the sponsor declassifies a VTA you created, the omission appears in your system. Declassifications occur only if you create a manual classification and the sponsor declassifies the verbatim term without reclassifying it.
Global VTA Conflicts
If the sponsor sends Global VTAs in the XML file, but your organization has set the TMS_DSI local reference codelist value GLOBALVTA to not accept Global VTAs, you will receive errors or warnings for those VTA records, depending on the setting of the codelist value, as follows:
-
Fail Global VTA (F). If GLOBALVTA is set to F, and the sponsor sends a Global VTA in the XML file, the system does not import the record. The record remains in the error file with an appropriate error.
-
Domain Only (D). If GLOBALVTA is set to D, and the sponsor sends a Global VTA in the XML file, the system creates a Domain VTA for each Global VTA sent, and associates a warning with the record in the error file.
If GLOBALVTA is set to Allow Global VTAs (A), the system imports any Global VTAs sent by the sponsor as Global VTAs, and you never receive Global VTA-related errors or warnings. For more information on GLOBALVTA, see Setting DSI Preferences with Reference Codelist Settings
Source Term Actions
The sponsor may have assigned an Action (either a global or a Discrepancy Message) to one or more omissions, indicating that the source term itself must be changed in the external system before it can be processed in TMS. You can see these Actions in the Classify VT Omissions window.
If you are using Oracle Clinical, these Actions are automatically transmitted to Oracle Clinical during the next Batch Validation. If you use a different external source data system, you must ensure that the Actions are sent to that system.
The import job associates a warning with the omission in the error file in the following situations:
-
The omission no longer exists in your system (because it has been manually classified at your site or deleted in the source system).
-
The omission exists but is currently owned by the external source data system (Oracle Clinical, for example). This would occur if someone assigned an Action to the omission at your site and sent it to the external system.
Missing Dictionary Terms
The import process compares the dictionary terms used in sponsor VTAs in the file to the dictionary terms on your site. If any dictionary terms required by the sponsor are missing on your site, you must add them, either individually (see Modifying Repository Data in the Maintain Repository Data Window) or by reloading the sponsor's version of the dictionary.
Your organization's setting of the TMS_DSI local reference codelist value OUTSYNCCSP determines how your system handles this situation. The options are:
-
OUTSYNCCSP=E (Fatal Error). The import job fails at the first such error it finds. The error file provides information on the missing dictionary term. You cannot import the remaining data in the file until you add the required dictionary term to your database. However, records that were successfully processed before the failed record remain changed in the database. The error file contains a record of each successfully processed record.
After you add the required dictionary term, you can remove the fatal error message from the XML error file, move the file to the import directory, and run the import process again to process the remaining records.
-
OUTSYNCCSP=W (Warning). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. (Unlike the behavior on the sponsor site, the system does not create an Action associated with an omission for the source term whose VTA's dictionary term is missing in the importing system.)
You must add the dictionary term manually to bring your dictionary into synchronization with the sponsor's, or reload the sponsor's version of the dictionary.
-
OUTSYNCCSP=N (Neither). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. There is no mention of the missing dictionary term in the error file.
For more information on OUTSYNCCSP, see Setting DSI Preferences with Reference Codelist Settings
Parent topic: Using DSI from the Source Site
Updating External System Data
You must run the necessary job in the external source data system to synchronize its data with the sponsor site's TMS repository.
If the source data system is Oracle Clinical, the Batch Validation job derives TMS values into Oracle Clinical.
Parent topic: Using DSI from the Source Site
Using DSI from the Sponsor Site
This section includes:
- Importing Data
- Processing Source Site Data
- Exporting Data
- Sending the XML File to the Source Site
- Removing X Area Data from the Sponsor Site
Parent topic: Using TMS with Disconnected Systems
Importing Data
A source site sends data in one or more XML files to your input directory for processing. Each XML file contains data for a single X Area. Typically, an X Area corresponds to a study; see Defining X Areas.
To import the data into your system, run the import job; from the VTA Maintenance menu, select Disconnected System Integration, then Import DSI Data (see Import DSI Data). The import job runs on all XML files in the input directory.
As part of the import process, Autoclassification runs on all omissions in the X Area, including those sent by the source site.
You can see the status of import files and data; from the VTA Maintenance menu, select Disconnected System Integration, then Maintain DSI Files. For instructions, see Viewing XML File and Record Statuses.
This section describes how the import process handles the following situations:
Adding New Source Terms
Any terms that have been updated (for example, through Oracle Clinical Data Entry Update) or added to TMS from the CRO's external source data system since the last source site data export/sponsor import are included in the XML file and imported by the system, associated with their VTAs, omissions, and associated Informative Notes, if any.
Deleting Source Terms
The source site may mark source terms for deletion if they have been deleted in the external source data system. The import process deletes these terms from your database as well.
Classification Conflicts
The import process compares the VTAs in the XML file with those for the same verbatim terms in your database. If a source site VTA conflicts with your VTA for the same verbatim term—the verbatim term is to a different dictionary term, or one partner has given the VTA a subtype of Misspelled and the other a subtype of Accepted—the system does not import the source site VTA. It remains in the error file, associated with a warning, and the system sets the verbatim term's Update box to Y in your database, forcing the record to be included in your next DSI data export XML file. When the source site imports that file, your VTA overwrites the source site's in the source site's database.
Global VTA Conflicts
If the source site sends Global VTAs in the XML file, but your organization has set the TMS_DSI local reference codelist value GLOBALVTA to not accept Global VTAs, you will receive warnings for those VTA records. The system either does not import the record or imports it as a Domain VTA, depending on the setting of the codelist value, as follows:
-
Fail Global VTA (F). If GLOBALVTA is set to F, and the source site sends a Global VTA in the XML file, the system does not import the record. The record remains in the error file associated with an appropriate error.
-
Domain Only (D). If GLOBALVTA is set to D, and the source site sends a Global VTA in the XML file, the system creates a Domain VTA for each Global VTA sent, and associates a warning with the record in the error file.
In either of the above cases, the system sets the record's Update box to Y in your database, forcing the record to be included in your next DSI Data Export XML file. When the source site imports that file, your VTA overwrites the source site's in the source site's database.
If GLOBALVTA is set to Allow Global VTAs (A), the system imports any Global VTAs sent by the source site as Global VTAs, and you never receive Global VTA-related errors or warnings.
For more information on GLOBALVTA, see: Setting DSI Preferences with Reference Codelist Settings.
Importing Approved and/or Unapproved VTAs
The setting of the TMS_DSI local reference codelist value CREVTA determines whether your system imports VTAs created by the source site as Approved, Unapproved, or some of each, as follows:
-
Unapproved (U). If CREVTA is set to U, the system imports all VTAs created by the source site as Unapproved, regardless of whether or not they were created as Approved at the source site. You must examine each one manually and either approve it or reclassify the verbatim term. See Approving a Nonapproved VTA and Reclassifying a Verbatim Term.
Note:
Even if you have this reference codelist set to U, the import job imports VTAs that are direct matches to dictionary terms—that is, VTAs that were created during Autoclassification at the source site—as Approved.
-
XML (X). If CREVTA is set to X, the system imports each VTA created by the source site with the same approval status given them at the source site. You must manually process only the Unapproved VTAs. This is the default setting.
-
Approved (A). If CREVTA is set to A, the system imports all VTAs created by the source site as Approved, regardless of whether or not they were approved at the source site. No manual processing is necessary. This setting is not recommended.
For more information on CREVTA, see: Setting DSI Preferences with Reference Codelist Settings.
Source Site-Created Actions
The source site may have assigned an external Action to one or more verbatim terms, indicating that the source term itself must be changed in the external system before it can be processed in TMS. These Actions are imported in association with an omission. You can see these Actions in the Classify VT Omissions window.
You can modify source site-created Actions only if the Action is currently owned by TMS. This can happen at two points:
-
when the source site has not yet sent the Action to the source data system (via Batch Validation, if the external system is Oracle Clinical)
-
when the external system sends the Action back to TMS with a comment or question
Missing Dictionary Terms
The import process compares the dictionary terms used in source site VTAs to the dictionary terms on your site. If any dictionary terms required by the source site are missing on your site, it indicates that the source site has added dictionary terms that your organization does not support. The dictionaries at the two sites must be re-synchronized.
Your organization's setting of the TMS_DSI local reference codelist value OUTSYNCCSP determines how your system handles this situation. The options are:
-
OUTSYNCCSP=E (Fatal Error). The import job fails at the first such error it finds. The error file provides information on the missing dictionary term. Records that were successfully processed before the failed record remain changed in the database. The error file contains a record of each successfully processed record.
Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.
-
OUTSYNCCSP=W (Warning). The import job succeeds, but VTAs whose dictionary terms are missing in your system are not imported. The system creates an Action associated with an omission for the source term whose VTA's dictionary term is missing in your system. This Action is visible in the Classify VT Omissions window, as well as in the error file. It is not sent to the external source data system, as other Actions are. The Action remains associated with the omission until it is resolved. You can also read the error file to find these VTAs and the dictionary term they require. Records without errors are successfully imported.
Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.
-
OUTSYNCCSP=N (Neither). The import job succeeds, and records without errors are successfully imported. Any VTA whose dictionary term is missing in your system is not imported and an omission is created for that VTA on your site. If an error file is created, it contains no mention of the missing dictionary term.
The system creates an Action associated with an omission for the source term whose VTA's dictionary term is missing in your system. This Action is visible in the Classify VT Omissions window. It is not sent to the external source data system, as other Actions are, or to the source site. The Action remains associated with the omission until it is resolved.
Call the source site, resolve the dictionary discrepancy, and ask the source site to reexport the whole X Area.
For more information on OUTSYNCCSP, see: Setting DSI Preferences with Reference Codelist Settings
Parent topic: Using DSI from the Sponsor Site
Processing Source Site Data
After running Autoclassification, you must manually process the following source site data. When you update a record, the system sets its Update box to Y, causing the next export job to extract the record to send back to the source site.
Omissions
You must manually classify, or assign Actions to, all the source terms still unclassified after Autoclassification. See Classifying Terms Manually.
Unapproved VTAs
You must examine each Unapproved VTA received from the source site and either approve it or reclassify the verbatim term.
Conflicting Classifications
In the error file, the import process associates warnings with all source site classifications that conflict with VTAs in your database. You can view the conflicts in the error file. The system also sets the Update box to Y for the relevant verbatim term in the database, forcing the record to be included in the next export XML file. During the next source site import, your VTA will automatically overwrite the source site's.
If VTAs conflict because the source site used a dictionary term that does not exist in your repository, the job may fail. See Missing Dictionary Terms.
Parent topic: Using DSI from the Sponsor Site
Exporting Data
When you have finished processing data from the source site, you must extract the updated data for the relevant X Area into an XML file and send it back to the source site. You can either export data from a single X Area (see Export DSI Data) or from all active X Areas at once (see Export Active X Areas). Both jobs export only changed data, using both timestamps and the Update setting to determine which records have been updated.
If necessary, you can force the export of all data by using the Force Rederivation job (see Force Rederivation) to set all records' Update boxes to Y. Then you must run either Export DSI Data or Export Active X Areas to export the data.
Data Exported
The data extract jobs automatically extract the following types of data in the required XML format:
-
All verbatim term omissions with new Actions
-
All VTAs for verbatim terms based on source terms from the receiving source site that have been created or changed since the last export, including classifications, reclassifications, and approvals
Note:
If the local reference codelist TMS_DSI parameter XAPPVTA is set to Y, the system exports only Approved VTAs. Unapproved VTAs are not exported.
-
New and changed status audit Informative Notes associated with verbatim terms
-
All VTAs declassified since the last export of the same X Area
Export Statuses
You can check on the export's status (see Viewing XML File and Record Statuses). Export jobs can have only two statuses: Success and Fatal Error. The export process should fail only if there is a problem with the Output directory.
Parent topic: Using DSI from the Sponsor Site
Sending the XML File to the Source Site
DSI does not handle transporting the XML file to the source site. You must do it manually; for example, zipping the file and sending it via either FTP or email.
To avoid sending the same files to the partner for unnecessary reprocessing, manually delete files from the output directory after sending them to the other site.
Parent topic: Using DSI from the Sponsor Site
Removing X Area Data from the Sponsor Site
Faulty data may be imported into the sponsor site. Therefore, if you have special privileges, you can delete all data belonging to a specific X Area. See Delete DSI X Areas. You can then reexport all X Area data from the source site (see Force Rederivation) and reimport the X Area.
Parent topic: Using DSI from the Sponsor Site
Viewing XML File and Record Statuses
From the Disconnected System Integration menu, select Maintain DSI Files.
In the Maintain DSI Files window you can view all the import and export jobs associated with each X Area, and see how many records in each file were successfully updated, updated with a warning, and failed. In addition, you can assign a manual status to a file to reflect the state of your work on the file, and view the error file for each job.
For more information, see:
Parent topic: Using TMS with Disconnected Systems
Viewing File and Record Status
The Maintain DSI Files window has two sections: X Areas and Files. After you execute a query, the system displays all the X Areas that meet the filter and query criteria in the X Areas section. In the Files section, the system displays all the files that meet the criteria for the X Area selected in the X Areas section.
To view the files for a different X Area, click its name in the X Area column. In the Files section, the system displays its files that meet the filter and query criteria. For each file, the system displays the following information:
-
File Name. The file name is generated by the system at the time of export or import (see XML File Names in Disconnected System Integration XML Files for information on the naming convention).
-
Process Type: Import or Export.
-
Process Status. The way the system determines the process status depends in part on the settings of the local reference codelist TMS_DSI. SeeUsing DSI from the Source Site and Using DSI from the Sponsor Site for the implications of the statuses on your site. The possible statuses are:
-
Success. All records were processed successfully, with no warnings or errors. There is no error file.
-
Warnings. All records were processed successfully, but at least one record has an error. You can see warnings associated with each record with an error in the error file. Depending on the OUTSYNCCSP reference codelist setting, there may be a discrepancy message associated with VTA records whose dictionary term is not included in your dictionary.
-
Errors. Some records may have been processed successfully, but at least one record failed import and remains in the error file associated with an error message.
-
Fatal. The job failed due to a problem with the directory or the XML file itself, or the source site and sponsor dictionaries were out of synch and OUTSYNCCSP was set to fail in that situation.
For more information on OUTSYNCCSP, see: Setting DSI Preferences with Reference Codelist Settings.
-
-
Manual Status. This status is meant to indicate the state of your work on files with warnings or errors, for use in querying. The system does not enforce any behavior related to the manual status. The possible statuses are:
-
New.
-
Pending.
-
Fixed.
-
-
# Source Data Records. The following three columns concern source term records that are new, updated, or deleted. To see warnings and errors, click View File or select Options, then choose View File.
-
OK: The number of source terms successfully processed without warnings.
-
Warning: The number of source terms successfully processed but with warnings.
-
Failed: The number of source terms with errors; these records were not successfully processed.
-
-
# VTA Data Records. This includes new verbatim term assignments (VTAs, or classifications) reclassified and declassified VTAs, and newly approved or unapproved VTAs. The following three columns concern records that are new VTAs:
-
OK: The number of VTAs successfully processed without warnings.
-
Warning: The number of VTAs successfully processed but with warnings. You can see the warnings in the error file.
-
Failed: The number of VTAs with errors; these records were not successfully processed. You can see the errors in the error file.
-
-
Process Start Time. The time the import or export process began.
-
Process End Time. The time the import or export process finished.
-
Process Elapsed. The total amount of time spent processing the file, in HH:MM:SS format.
-
Audit Information. The following three fields contain audit information for the file selected above:
-
Created By
-
Creation Time
-
Modified By. The user ID of the person who last changed the manual status.
-
Modification Time. The timestamp of the last manual status change.
-
-
Filter. The system displays the current filter settings.
Parent topic: Viewing XML File and Record Statuses
Deleting Files
You can delete X Area files from view by selecting Record, then Delete. This does not delete the file itself, only the database information on the file. You can no longer view the file from this window.
Parent topic: Viewing XML File and Record Statuses
Setting the Filter
The filter determines which X Areas and files are displayed in the Maintain DSI Files window. Your default filter settings are set by selecting the Definition menu, then Maintain Settings.
You can reset the filter at any time by clicking the Filter button in the Maintain DSI Files window. Enter values in the filter pop-up box to limit the files displayed in the Maintain DSI Files window. By default all files are displayed. All settings are optional.
-
External System. From the list, select the external system where the source data was originally collected.
-
Source Data DB. From the list, select the database of the external system where the source data was originally collected.
-
X Area and X Area Name. From the list, select the X Area. The list of values in the X Area field includes both the X Area ID and name, and when you select a value, the system populates both fields.
-
X Area Status. From the list, choose the current status of the X Area(s) for the file(s) you want to see. The choices are Active, Inactive, and Complete (for an explanation of the statuses, see Defining X Areas).
-
Last Export Between. Enter the start and end dates of the period in which jobs you want to see were executed. Use the format specified for your installation in OPA Settings (OPA_SQL_DATE_FORMAT); for example, if you use the format DD-MON-YYYY, enter 15-OCT-2004.
-
Manual Status. From the list, select the manually set status of the file(s) you want to see. The choices are: New, Pending, and Fixed.
-
Process Type. From the list, select the process that produced the file you want to see; either Export or Import.
-
Process Between…and.… Enter the start and end dates of the period in which jobs you want to see were executed. The system returns all files that were even partially executed during that data range. Use the format specified for your installation in OPA Settings (OPA_SQL_DATE_FORMAT); for example, if you use the format DD-MON-YYYY, enter 15-OCT-2004.
-
Process Status. Select one or more status. The system will display files resulting from jobs with the statuses you select, and that meet the other criteria.
Click one of the following buttons:
-
Clear. Returns values to what they were when you opened the filter.
-
Restore. Resets the filter to the default values (from the Definition menu, select Maintain Settings to set).
-
OK. Saves the current settings. The filter pop-up window disappears, leaving the Maintain DSI Files window in view. You must execute a query (from the Query menu, select Execute Query or press F8) to display the files that meet the filter criteria. The filter criteria are displayed at the bottom of the Files section of the window. These settings persist the next time you open the Maintain DSI Files window.
-
Cancel. Closes the Filter window without saving any changes.
Parent topic: Viewing XML File and Record Statuses
Viewing Error Files
In the Maintain DSI Files window under the VTA Maintenance menu, you can see the results of each DSI export and import job. If a job had errors, you can look at the error file to find the errors and then fix them and rerun the job.
To find a job, do the following:
The file is in XML format. It is the same XML file that was imported or exported, except that a warning or error is displayed with each record that has a warning or error.
Parent topic: Using TMS with Disconnected Systems
DSI Batch Jobs
Note:
Only superusers can run DSI batch jobs. For information on setting up a superuser account, see Defining a New User and also Running the DSI Initialization Job.
This section includes information on DSI batch jobs:
- Export DSI Data
- Import DSI Data
- Export Active X Areas
- Export X Areas by External System/Instance
- Import X Areas by External System/Instance
- Delete DSI X Areas
- Delete Remote DSI Databases
- Force Rederivation
Parent topic: Using TMS with Disconnected Systems
Export DSI Data
This job extracts data for a single X Area into an XML file located in the Output directory. It processes only data that has been added, deleted, or updated since the last export job was run ("incremental" data processing). The export process should fail only if there is a problem with the receiving output directory. To run the job:
Parent topic: DSI Batch Jobs
Import DSI Data
The import process operates on all XML files currently in the input directory, and deletes them from that directory upon completion. If there are errors or warnings, or the job fails, the system inserts the errors into the XML file associated with the appropriate records, and moves the XML file to the error directory.
The import process begins by calling the XML parser to validate the structure of the XML file against the XML schema. If it is invalid, the job fails. The import job also validates the XML metadata against the mappings defined in the sponsor.
The data import job operates on data differently at the source site and the sponsor site; see Using DSI from the Source Site, and Using DSI from the Sponsor Site. The way it handles different errors depends on the settings of several reference codelist values in the importing system; see Setting DSI Preferences with Reference Codelist Settings.
To run the job:
- Go to Disconnected System Integration, then select Import DSI Data.
- Enter the name of the report server you want to run the job.
- Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.
Import Process Statuses
Import jobs can end with four possible statuses: Success, Warning, Error, and Fatal Error. Jobs that end with a status of Success do not have error files. Jobs ending with all other statuses do have error files.
To create the error file, the import process inserts errors or warnings, associated with the appropriate record, into the XML file in the input directory and then moves the XML file into the error directory.
-
Success: If all records are imported without errors or warnings, DSI gives the job a status of Success, updates the timestamp of each imported record in the importing database, and deletes the XML file from the input directory. It does not produce an error file.
-
Warning: If the import process encounters data errors or conflicts, and you have set up DSI to continue even with errors (see Setting DSI Preferences with Reference Codelist Settings), the import job:
-
successfully imports all records that do not have warnings, and removes those records from the XML file
-
for records with errors or conflicts, does one of the following, depending on reference codelist settings:
-
does not import the record; leaves it in the error file associated with the appropriate error
-
imports the record but associates a warning with the record in the error file; in addition, may associate a Discrepancy Message with the record
-
-
-
Error: If the import process encounters data errors or conflicts, and you have set up DSI to fail in that situation (see Setting DSI Preferences with Reference Codelist Settings), the import job:
-
does not import records with errors, but keeps them in the error file associated with an error message
-
imports records successfully before encountering the error but also leaves these imported records, and the unprocessed ones listed after the record that caused the error, in the file
-
-
Fatal Error: The following types of errors are fatal and always result in job failure:
-
An input directory does not exist in the importing system, or an output directory does not exist on the exporting system.
-
(Import only) The XML file structure is invalid.
-
(Import only) The dictionary required in the XML file does not exist in the importing system.
-
(Import only) The Oracle database user has not been given Delete privileges for the input directory.
-
(Import only) A new VTA is created using a dictionary term that does not exist in the importing system.
-
(Import only) The input XML file contains the wrong external system name or instance name for the specified X Area.
The error file includes information about the fatal error and all the records originally contained in the file.
-
Parent topic: DSI Batch Jobs
Export Active X Areas
This job extracts data for all active X Areas, creating a separate XML file in the Output directory for each active X Area. It processes only data that has been added, deleted, or updated since the last export job was run ("incremental" data processing). The export process should fail only if there is a problem with the receiving output directory.
- Go to Disconnected System Integration, then select Export DSI Data.
- Enter the name of the report server you want to run the job.
- Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.
Parent topic: DSI Batch Jobs
Export X Areas by External System/Instance
This job allows you to export data for a particular X Area from a single site. For example, a sponsor may want to export data from a single CRO rather than all CRO participating in the same study (X Area) at the same time.
To run the job:
- Go to Disconnected System Integration, then select Export X Areas by External System/Instance.
- From the External System drop-down list, select the external system to which you want to export data.
- From the Instance drop-down list, select the database to which you want to export data.
- Enter the name of the report server you want to run the job.
- Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Scheduling Parameters.
Parent topic: DSI Batch Jobs
Import X Areas by External System/Instance
This job allows you to import data for a particular X Area from a single site. For example, a sponsor may want to import data from a single CRO rather than all CRO participating in the same study (X Area) at the same time.
To run the job:
- Go to Disconnected System Integration, then select Import X Areas by External System/Instance.
- From the External System drop-down list, select the external system from which you want to import data.
- From the Instance drop-down list, select the database from which you want to import data.
- Enter the name of the report server you want to run the job.
- Run the job. You can schedule this job to be run on a regular basis such as nightly or weekly; see Running a Job.
Parent topic: DSI Batch Jobs
Delete DSI X Areas
Available on the sponsor site only. If faulty data has been loaded into the sponsor site or exported from the source site, it may be necessary to remove all data belonging to the X Area and start again. This job deletes all data belonging to a single X Area: the X Area definition, source terms, omissions, and the database records of the X Area files (not the files themselves).
To run the job:
-
Go to VTA Maintenance, select Disconnected System Integration, then Delete X Area.
-
Enter the following parameters:
-
External System. The name of the external system where the source data originated.
-
Instance. The database where the source data originated.
-
X Area. The X Area that corresponds to the unit whose data you want to export; typically a study or case.
-
Report Server Name. Enter the name of the report server you want to run the job.
-
If data has become corrupted on the sponsor site, you can export all relevant data from a particular X Area to the other system.
- Delete the X Area on the sponsor site by running the Delete DSI X Areas job on the sponsor.
- On the source site, run Force Rederivation on the same X Area.
- On the source site, run Export DSI Data on the same X Area.
- Send the data from the source site to the sponsor.
- Run Import DSI Data on the sponsor site to recreate the X Area and all its data.
Parent topic: DSI Batch Jobs
Delete Remote DSI Databases
To delete a registered remote database from a sponsor site, do the following:
-
Manually delete all DSI definitions to the remote database in the Define DSIs window, using Delete from the Record menu. This has the effect of deleting all the associated X Areas as well.
-
Run the Delete Remote DSI Databases job.
To run the job:
- In the Definitions menu, click Jobs and then click Delete Remote DSI Databases. The job window opens.
- In the job-specific parameter Remote Instance field, select the name of the remote instance from the drop-down list. The system displays only remote instances that are not part of a DSI definition.
- Run the job. See Running a Job.
Parent topic: DSI Batch Jobs
Force Rederivation
This job marks all source terms and omissions in a single X Area for processing in the next export process, whether or not they have been added or modified since the last export job ("full" data processing). To actually process the data, you must run the Export DSI Data job.
If data has become corrupted on the other site, or dictionaries are no longer synchronized, you can export all relevant data from a particular X Area to the other system.
- From the Definition menu, select Define Disconnected System Integrations, then X Areas, and click the Force Rederivation button. The system marks for mandatory processing (Update=Y) all data in this X Area of the types normally exported from the system (which vary, depending on whether the exporting system is a sponsor or a source site).
- From the DSI Maintenance menu, select Export DSI Data and run the job; see Export DSI Data.
Parent topic: DSI Batch Jobs